

Yuan2.0 M32
Overview :
Yuan2.0-M32 is a mixed expert (MoE) language model featuring 32 experts, out of which 2 are active. It introduces a novel routing network—attention routing—to improve expert selection efficiency, achieving a 3.8% increase in accuracy. The model is trained from scratch using 2000B tokens, with a training computational load only 9.25% of that required by a dense model with the same parameter scale. It demonstrates competitive performance in coding, mathematics, and various specialized fields, utilizing just 3.7B active parameters, with a per-token forward computation requirement of only 7.4 GFLOPS, which is 1/19 of what Llama3-70B demands. It surpasses Llama3-70B in MATH and ARC-Challenge benchmark tests, achieving accuracy rates of 55.9% and 95.8%, respectively.
Target Users :
Yuan2.0-M32 is designed for developers and researchers who require efficient computation and reasoning in coding, mathematics, and specialized fields. Its low computational demand and high accuracy make it an ideal choice for large-scale language model applications.
Use Cases
Used for developing natural language understanding applications.
Provides precise computational support in solving mathematical problems.
Serves as an assistive tool for knowledge acquisition and reasoning in specialized fields.
Features
Mixed expert (MoE) model with 32 experts, 2 of which are active.
Utilizes a new attention routing network to enhance the efficiency of expert selection.
Trained from scratch using 2000B tokens with a low training computational demand.
Excels in coding, mathematics, and specialized fields with competitive performance.
Outperforms other models in MATH and ARC-Challenge benchmark tests.
Operates with just 3.7B active parameters, demonstrating high computational efficiency.
How to Use
1. Set up the environment; it's recommended to use the latest Docker image of Yuan2.0-M32.
2. Preprocess the data according to the provided scripts.
3. Use the sample scripts for model pre-training.
4. Follow the detailed deployment plan from vllm for inference service deployment.
5. Check the GitHub repository for more information and documentation.
Featured AI Tools

Gemini
Gemini is the latest generation of AI system developed by Google DeepMind. It excels in multimodal reasoning, enabling seamless interaction between text, images, videos, audio, and code. Gemini surpasses previous models in language understanding, reasoning, mathematics, programming, and other fields, becoming one of the most powerful AI systems to date. It comes in three different scales to meet various needs from edge computing to cloud computing. Gemini can be widely applied in creative design, writing assistance, question answering, code generation, and more.
AI Model
11.4M
Chinese Picks

Liblibai
LiblibAI is a leading Chinese AI creative platform offering powerful AI creative tools to help creators bring their imagination to life. The platform provides a vast library of free AI creative models, allowing users to search and utilize these models for image, text, and audio creations. Users can also train their own AI models on the platform. Focused on the diverse needs of creators, LiblibAI is committed to creating inclusive conditions and serving the creative industry, ensuring that everyone can enjoy the joy of creation.
AI Model
6.9M