# Mixture of Experts Model

Moonlight is a 16B parameter Mixture of Experts (MoE) model trained using the Muon optimizer, demonstrating outstanding performance in large-scale training. By incorporating weight decay and adjusting parameter update ratios, it significantly enhances training efficiency and stability. This model surpasses existing models in various benchmarks while substantially reducing the computational resources required for training. Moonlight's open-source implementation and pre-trained models provide researchers and developers with a powerful toolset, supporting diverse natural language processing tasks such as text generation and code generation.

DeepSeek-VL2 is a series of advanced large-scale mixture of experts (MoE) visual language models, significantly improved compared to its predecessor DeepSeek-VL. This model series demonstrates exceptional capabilities across various tasks, including visual question answering, optical character recognition, document/table/chart understanding, and visual localization. Comprising three variants: DeepSeek-VL2-Tiny, DeepSeek-VL2-Small, and DeepSeek-VL2, with 1 billion, 2.8 billion, and 4.5 billion active parameters respectively, DeepSeek-VL2 achieves competitive or state-of-the-art performance against existing dense and MoE-based open-source models, even with a similar or fewer number of active parameters.

Aria is a multimodal native mixture of experts model that excels in multimodal, language, and coding tasks. It performs exceptionally well in video and document understanding, supporting up to 64K multimodal input, with the ability to describe a 256-frame video in just 10 seconds. The model has 25.3 billion parameters and can be loaded on a single A100 (80GB) GPU using bfloat16 precision. Aria was developed to meet the needs for multimodal data understanding, particularly in video and document processing. It is an open-source model aimed at advancing multimodal artificial intelligence.

GRIN-MoE is a Mixture of Experts (MoE) model developed by Microsoft, focusing on enhancing performance in resource-limited environments. By employing SparseMixer-v2 to estimate the gradient for expert routing, GRIN-MoE achieves model training scalability without relying on expert parallel processing or token dropping, unlike traditional MoE training methods. It excels particularly in coding and mathematical tasks, making it suitable for scenarios that demand strong reasoning capabilities.

Yuan2.0-M32-hf-int8 is a mixture of experts (MoE) language model featuring 32 experts, of which 2 are active. By adopting a new routing network—the attention router—it enhances the efficiency of expert selection, resulting in an accuracy improvement of 3.8% compared to models using traditional routing networks. Yuan2.0-M32 was trained from scratch on 200 billion tokens, with its training computation demand being just 9.25% of that required by a dense model of equivalent parameter size. This model is competitive in programming, mathematics, and various specialized fields while utilizing only 3.7 billion active parameters, which is a small portion of a total of 4 billion parameters. The forward computation per token requires only 7.4 GFLOPS, just 1/19th of what Llama3-70B demands. Yuan2.0-M32 outperformed Llama3-70B in the MATH and ARC-Challenge benchmark tests, achieving accuracy rates of 55.9% and 95.8%, respectively.