# MoE

Flock of Finches 37B-A11B v0.1 is the latest member of the RWKV family, representing an experimental model with 1.1 billion active parameters. Despite being trained on only 109 billion tokens, it performs comparably to the recently released Finch 14B model on common benchmark tests. This model employs an efficient sparse mixture of experts (MoE) approach, activating only a portion of parameters for any given token, thereby saving time and reducing computational resource usage during training and inference. Although this architectural choice incurs higher VRAM usage, from our perspective, it is highly beneficial to train and operate a model with greater capacity at a lower cost.

Skywork-MoE is a 146-billion parameter high-performance Mixture of Experts (MoE) model, featuring 16 experts and 2.2 billion activation parameters. This model is initialized from the dense checkpoint of the Skywork-13B model. Two innovative techniques are introduced: gated logic normalization, enhancing expert diversity; and adaptive auxiliary loss coefficient, allowing layer-specific auxiliary loss coefficient adjustment. Skywork-MoE demonstrates comparable or superior performance to models with more parameters or activation parameters across various popular benchmark tests, such as C-Eval, MMLU, CMMLU, GSM8K, MATH, and HumanEval.

Qwen1.5-MoE-A2.7B is a large-scale MoE (Mixture of Experts) language model with only 2.7 billion activation parameters. Despite its smaller size, it achieves performance comparable to 70 billion parameter models. Compared to traditional large models, this model reduces training costs by 75% and increases inference speed by 1.74 times. It employs a special MoE architecture design, including fine-grained experts, new initialization methods, and routing mechanisms, which significantly enhance model efficiency. This model is suitable for various tasks in natural language processing, code generation, and more.