# Mixed Expert

Yuan2.0-M32 is a mixed expert (MoE) language model featuring 32 experts, out of which 2 are active. It introduces a novel routing network—attention routing—to improve expert selection efficiency, achieving a 3.8% increase in accuracy. The model is trained from scratch using 2000B tokens, with a training computational load only 9.25% of that required by a dense model with the same parameter scale. It demonstrates competitive performance in coding, mathematics, and various specialized fields, utilizing just 3.7B active parameters, with a per-token forward computation requirement of only 7.4 GFLOPS, which is 1/19 of what Llama3-70B demands. It surpasses Llama3-70B in MATH and ARC-Challenge benchmark tests, achieving accuracy rates of 55.9% and 95.8%, respectively.

DeepSeek-V2 is a mixed expert (MoE) language model consisting of 236B parameters, activated with 21B parameters per token. While maintaining cost-efficient training and efficient inference, it activates each token with 21B parameters. Compared to the previous DeepSeek 67B, DeepSeek-V2 offers superior performance while saving 42.5% of training costs, reducing 93.3% of KV cache, and increasing the maximum generation throughput by 5.76 times. The model has been pretrained on an 8.1 trillion token high-quality corpus and further optimized through supervised fine-tuning (SFT) and reinforcement learning (RL), performing exceptionally well in standard benchmark tests and open-source generation evaluations.

MistralAI's new 8x7B mixed-expert (MoE) base model for text generation. This model utilizes a mixed expert architecture to produce high-quality text. Its advantages include generating high-quality text and being applicable to various text generation tasks. Pricing is determined based on usage; please refer to the official website for details. This model aims to address challenges in text generation tasks.