# Long Context

Qwen2.5-Coder is the latest series of the Qwen large language models, focusing on code generation, reasoning, and debugging. Based on the powerful Qwen2.5 architecture, this model was trained on 550 trillion source codes, text-code associations, synthetic data, and more, making it a leader among current open-source code language models. It not only enhances programming capabilities but also retains advantages in mathematics and general-purpose tasks.

The Qwen2.5-Coder-3B-Instruct-GPTQ-Int8 is a large language model optimized for code generation, reasoning, and debugging, part of the Qwen2.5-Coder series. Based on Qwen2.5, it has been trained on a dataset including source code, code-text associations, and synthetic data, achieving 550 trillion training tokens. The Qwen2.5-Coder-32B has emerged as the leading open-source large language model for code, matching coding capabilities with GPT-4o. This model also provides a comprehensive foundation for real-world applications such as code assistance, augmenting coding capabilities while maintaining strengths in mathematics and general skills.

LongVA is a long context transformer model capable of processing over 2000 frames or 200K visual tokens. It achieves leading performance in Video-MME among 7B models. The model is tested on CUDA 11.8 and A100-SXM-80G and can be quickly deployed and used through the Hugging Face platform.

Samba is a simple yet powerful hybrid model with infinite context length. Its architecture is straightforward: Samba = Mamba + MLP + Sliding Window Attention + Hierarchical MLP Stacking. The Samba-3.8B model was trained on the Phi3 dataset with 32 trillion tokens and significantly outperformed Phi3-mini on major benchmark tests (e.g., MMLU, GSM8K, and HumanEval). Samba can also achieve perfect long-context retrieval ability with minimal instruction tuning while maintaining linear complexity with respect to the sequence length. This enables Samba-3.8B-instruct to excel in downstream tasks such as long-context summarization.