# Long Text Processing

GPT-4.1 is a series of new models that offer significant performance improvements, particularly in coding, instruction following, and handling long text contexts. Its context window has been expanded to 1 million tokens, and it excels in real-world applications, making it suitable for developers to create more efficient applications. This model is relatively low-cost and offers fast response times, making it more efficient for developing and executing complex tasks.

HunYuan T1 is an ultra-large-scale reasoning model based on reinforcement learning. Post-training significantly improves reasoning ability and aligns with human preferences. This model focuses on long-text processing and complex reasoning tasks, exhibiting significant performance advantages.

QwQ-32B is a reasoning model from the Qwen series, focusing on the ability to think and reason through complex problems. It excels in downstream tasks, especially in solving difficult problems. Based on the Qwen2.5 architecture, it has been optimized through pre-training and reinforcement learning, boasting 32.5 billion parameters and supporting a context length of up to 131,072 tokens. Its main advantages include powerful reasoning capabilities, efficient long-text processing capabilities, and flexible deployment options. This model is suitable for scenarios requiring deep thinking and complex reasoning, such as academic research, programming assistance, and creative writing.

MoBA (Mixture of Block Attention) is an innovative attention mechanism specifically designed for large language models dealing with long text contexts. It achieves efficient long sequence processing by dividing the context into blocks and allowing each query token to learn to focus on the most relevant blocks. MoBA's main advantage is its ability to seamlessly switch between full attention and sparse attention, ensuring performance while improving computational efficiency. This technology is suitable for tasks that require processing long texts, such as document analysis and code generation, and can significantly reduce computational costs while maintaining high model performance. The open-source implementation of MoBA provides researchers and developers with a powerful tool, driving the application of large language models in long text processing.

ModernBERT-base is a modern bidirectional encoder Transformer model pretrained on 2 trillion English and code samples, natively supporting up to 8192 tokens of context. The model incorporates cutting-edge architectural improvements such as Rotary Positional Embeddings (RoPE), Local-Global Alternating Attention, and Unpadding, showing exceptional performance on long-text processing tasks. It is ideal for processing long documents for tasks such as retrieval, classification, and semantic search within large corpuses. Since the training data is primarily in English and code, its performance may be reduced when handling other languages.

EXAONE 3.5 is a series of artificial intelligence models released by LG AI Research, renowned for their outstanding performance and cost-effectiveness. They excel in training efficiency, contamination mitigation, long text comprehension, and instruction-following abilities. The development of the EXAONE 3.5 model adheres to LG's AI ethics principles and includes an AI ethical impact assessment to ensure responsible usage. The release of these models aims to advance AI research and the ecosystem while laying a foundation for AI innovations.

Qwen2.5-Turbo is an innovative language model developed by Alibaba's team, optimized for processing extremely long texts. It supports a context of up to 1 million tokens, which is equivalent to approximately 1 million English words or 1.5 million Chinese characters. The model achieved a 100% accuracy rate in the 1M-token Passkey Retrieval task and scored 93.1 in the RULER long text evaluation benchmarks, surpassing both GPT-4 and GLM4-9B-1M. Qwen2.5-Turbo not only excels in long text handling but also maintains high performance in short text processing, offering exceptional cost-effectiveness at only 0.3 yuan per million tokens processed.

Qwen2.5-Coder is the latest series in the Qwen large language model, specifically designed for code generation, inference, and debugging. The model is based on Qwen2.5, extending the training tokens to 5.5 trillion, incorporating source code, textual code foundations, synthetic data, etc. Qwen2.5-Coder-32B stands out as a top performer among open-source code LLMs, matching the encoding capabilities of GPT-4o. This model is a GPTQ-quantized 4-bit instruction-tuned 3B parameter Qwen2.5-Coder model, featuring causal language modeling, pre-training and post-training phases, and a transformers architecture.

Qwen2.5-Coder-32B-Instruct-GPTQ-Int8 is a large language model specifically optimized for code generation within the Qwen series, featuring 3.2 billion parameters and supporting long text processing. It is one of the most advanced models in the field of open-source code generation. The model has been further trained and optimized based on Qwen2.5, showing significant improvements in code generation, inference, and debugging, while also maintaining strengths in mathematics and general capabilities. It utilizes GPTQ 8-bit quantization technology to reduce model size and enhance operational efficiency.

The Qwen2.5-Coder-32B-Instruct-GPTQ-Int4 is a large language model based on Qwen2.5, featuring 3.25 billion parameters and supporting long text processing with a maximum of 128K tokens. This model has shown significant improvements in code generation, code inference, and code repair, making it a leader among current open-source code language models. It not only enhances coding capabilities but also maintains strengths in mathematics and general reasoning.

Qwen2.5-Coder represents a series of large language models optimized for code generation, covering six mainstream model sizes with 0.5, 1.5, 3, 7, 14, and 32 billion parameters, catering to the diverse needs of developers. Qwen2.5-Coder shows significant improvements in code generation, inference, and debugging, trained on a robust Qwen2.5 backbone with a token expansion to 5.5 trillion, including source code, text grounding, and synthetic data, making it one of the most advanced open-source code LLMs, with coding capabilities comparable to GPT-4o. Additionally, Qwen2.5-Coder offers a more comprehensive foundation for applications in real-world scenarios such as code agents.

Qwen2.5-Coder-32B is a code generation model based on Qwen2.5, featuring 32 billion parameters, making it one of the largest open-source code language models available today. It shows significant improvements in code generation, reasoning, and fixing, capable of handling long texts up to 128K tokens, which is suitable for practical applications such as code assistants. The model also maintains advantages in mathematical and general capabilities, supporting long text processing, thus serving as a powerful assistant for developers in code development.

Mistral-Small-Instruct-2409 is an instruct-tuned AI model developed by the Mistral AI Team, featuring 22 billion parameters. It supports multiple languages and can handle sequences up to 128k in length. This model is particularly well-suited for scenarios that require long text processing and complex instruction understanding, such as in natural language processing and machine learning.

Reader-LM is a compact language model developed by Jina AI, designed to transform raw, messy HTML content from the web into clean Markdown format. These models are specifically optimized for long-text handling, support multiple languages, and can process context lengths of up to 256K tokens. By providing a direct conversion from HTML to Markdown, Reader-LM reduces reliance on regular expressions and heuristic rules, thereby enhancing conversion accuracy and efficiency.

InternLM-XComposer2.5 is a large language model specializing in text-image understanding and synthesis applications. With 7B parameters and support for 96K long text context, it is capable of handling complex tasks requiring extensive input and output.

InternLM2.5-7B-Chat-1M is an open-source 7 billion parameter dialogue model with excellent reasoning capabilities, outperforming models of similar size in mathematical reasoning tasks. It supports a 1M ultra-long context window, allowing it to handle long-text tasks like LongBench. Additionally, it boasts powerful tool-calling abilities, enabling it to gather information from hundreds of web pages for analysis and reasoning.

InternLM2.5-7B-Chat is an open-source 7 billion parameter Chinese dialogue model designed for practical scenarios. It boasts excellent reasoning abilities and surpasses models like Llama3 and Gemma2-9B in mathematical reasoning. It can analyze and reason from information gathered across hundreds of web pages, possesses strong tool invocation capabilities, supports a 1M ultra-long context window, making it suitable for building intelligent agents for long text processing and complex tasks.

The Llama-3 70B Gradient 524K Adapter is an extension of the Llama-3 70B model, developed by the Gradient AI Team. It is designed to extend the model's context length to over 524K through LoRA technology, thereby enhancing the model's performance in handling long text data. The model employs advanced training technologies, including NTK-aware interpolation and the RingAttention library, to efficiently train within high-performance computing clusters.

Llama-3 70B Instruct Gradient 1048k is an advanced language model developed by the Gradient AI team. By extending the context length to over 1048K, it demonstrates that SOTA (State of the Art) language models can learn to process long text after appropriate adjustments. The model employs NTK-aware interpolation and RingAttention technology, along with the EasyContext Blockwise RingAttention library, to efficiently train on high-performance computing clusters. It has widespread application potential in commercial and research applications, especially in scenarios requiring long text processing and generation.

Unichat-llama3-Chinese is the first Chinese instruction fine-tuning model based on the Meta Llama 3 model released by China Unicom's AI Innovation Center. By training with additional Chinese data, it has achieved high-quality Chinese question and answer capabilities, supporting up to 28K context input and planning to release a version supporting up to 64K. The fine-tuning instruction data has been manually screened to ensure high data quality. Additionally, the model plans to gradually release a 70 billion parameter Chinese fine-tuning version, including long text versions and versions with additional Chinese二次 pre-training.

Llama-3 8B Instruct 262k is a text generation model developed by the Gradient AI team, extending the context length of Llama-3 8B to over 160K and demonstrating the potential of state-of-the-art large language models in handling long text. This model achieves efficient learning on long texts through proper adjustment of the RoPE theta parameter, combined with NTK-aware interpolation and data-driven optimization techniques. Additionally, it is built upon the EasyContext Blockwise RingAttention library to support scalable and efficient training on high-performance hardware.