SigLIP2 is a multilingual vision-language encoder developed by Google, featuring improved semantic understanding, localization, and dense features. It supports zero-shot image classification, enabling direct image classification via text descriptions without requiring additional training. The model excels in multilingual scenarios and is suitable for various vision-language tasks. Key advantages include efficient image-text alignment, support for multiple resolutions and dynamic resolution adjustment, and robust cross-lingual generalization capabilities. SigLIP2 offers a novel solution for multilingual visual tasks, particularly beneficial for scenarios requiring rapid deployment and multilingual support.

This GitHub repository contains training and inference code to replicate our winning solution in the AI Mathematics Olympic (AIMO) Progress Prize 1. Our solution consists of four main parts: a recipe for fine-tuning DeepSeekMath-Base 7B for use in solving math problems using Tool Integrated Reasoning (TIR); two high-quality datasets of about 10 million math questions and solutions; an algorithm for generating solution candidates with coding execution feedback (SC-TIR); and four carefully selected validation sets from AMC, AIME, and MATH to guide model selection and avoid overfitting the public leaderboard.

Large language models increasingly rely on distributed techniques for training and inference. These techniques necessitate communication between devices, and as the number of devices increases, this can degrade scaling efficiency. While some distributed techniques can overlap communication to hide independent computation, techniques like tensor parallelism (TP) inherently serialize communication with model execution. One way to hide this serialized communication is to interweave it with producer operations (data generation) in a fine-grained manner. However, implementing this fine-grained communication and computation interleaving in software can be challenging. Furthermore, like any concurrent execution, it requires sharing computational and memory resources between computation and communication, leading to resource contention and decreased overlap efficiency. To overcome these challenges, we propose T3, which uses hardware-software co-design to transparently overlap serialized communication while minimizing resource contention with computation. T3, through simple configuration of producer output address spaces, transparently fuses producer operations and subsequent communication, requiring minimal software changes. At the hardware level, T3 incorporates lightweight tracking and triggering mechanisms to orchestrate producer computation and communication. It further leverages enhanced compute memory for computation related to communication. Consequently, T3 reduces resource contention and effectively overlaps serialized communication with computation. For important Transformer models like T-NLG, T3 achieves a geometric mean speedup of 30% (up to 47%) for communication-intensive sublayers and a geometric mean reduction of 22% (up to 36%) in data movement. Furthermore, T3's benefits persist as models scale: achieving a geometric mean speedup of 29% for sublayers in the 500B parameter sim model, PALM, and MT-NLG.

Zero Bubble Pipeline Parallelism is a crucial component of large-scale distributed training, and its efficiency is affected by pipeline bubbles. We introduce a scheduling strategy that successfully achieves zero pipeline bubbles under synchronous training semantics. The core idea behind this improvement is to divide backward calculation into two parts: one part calculates the gradients of the input, and the other part calculates the gradients of the parameters. Based on this idea, we manually designed novel pipeline scheduling, which significantly outperforms benchmark methods. We further developed an algorithm that automatically finds the optimal scheduling based on specific model configuration and memory constraints. Furthermore, to truly achieve zero bubbles, we introduce a novel technique that bypasses synchronization during optimizer steps. Experimental evaluation demonstrates that our method achieves up to 23% higher throughput than the 1F1B schedule under similar memory constraints. This number can further increase to 31% when memory constraints are relaxed. We believe our results mark an important step towards realizing the potential of pipeline parallelism.

ReFT is a simple yet effective method for enhancing the reasoning capabilities of large language models (LLMs). It first preheats the model through supervised fine-tuning (SFT), and then further fine-tunes the model using online reinforcement learning, specifically the PPO algorithm presented in this paper. ReFT significantly outperforms SFT by automatically sampling a large number of reasoning paths for a given problem and naturally deriving rewards from the true answers. ReFT's performance can be further improved by combining reasoning strategies (such as majority voting and re-ranking). It's noteworthy that ReFT achieves improvements by learning from the same training questions as SFT, without relying on additional or enhanced training questions. This demonstrates ReFT's stronger generalization ability.

This is an efficient LLM inference solution implemented on Intel GPUs. By simplifying the LLM decoder layer, utilizing segment KV caching strategies, and implementing a custom Scaled-Dot-Product-Attention kernel, this solution achieves up to 7x lower token latency and 27x higher throughput on Intel GPUs compared to the standard HuggingFace implementation. For detailed features, advantages, pricing, and positioning information, please refer to the official website.

BAGEL is a scalable unified multi-modal model that is revolutionizing the way AI interacts with complex systems. The model has dialogue reasoning, image generation, editing, style transfer, navigation, composition, thinking, and other functions, which provide a foundation for generating high-fidelity and realistic images by pretraining on large-scale alternating video and web data.

Aya Vision is an advanced visual model developed by the Cohere For AI team, focusing on multilingual and multimodal tasks and supporting 23 languages. The model significantly improves the performance of visual and text tasks through innovative algorithmic breakthroughs such as synthetic annotation, multilingual data augmentation, and multimodal model fusion. Its main advantages include efficiency (performing well even with limited computing resources) and extensive multilingual support. The release of Aya Vision aims to advance the forefront of multilingual and multimodal research and provide technical support to the global research community.

SigLIP2 is a multilingual vision-language encoder developed by Google, featuring improved semantic understanding, localization, and dense features. It supports zero-shot image classification, enabling direct image classification via text descriptions without requiring additional training. The model excels in multilingual scenarios and is suitable for various vision-language tasks. Key advantages include efficient image-text alignment, support for multiple resolutions and dynamic resolution adjustment, and robust cross-lingual generalization capabilities. SigLIP2 offers a novel solution for multilingual visual tasks, particularly beneficial for scenarios requiring rapid deployment and multilingual support.

The Intel? Core? Ultra 200 series desktop processors are the first AI PC processors designed for the desktop platform, delivering exceptional gaming experiences and industry-leading computing performance while significantly reducing power consumption. These processors feature up to 8 next-generation performance cores (P-cores) and up to 16 next-generation efficiency cores (E-cores), resulting in up to a 14% performance improvement in multi-threaded workloads compared to the previous generation. They are also the first desktop processors equipped with a neural processing unit (NPU) for enthusiasts, and include built-in Xe GPU technology, supporting advanced media features.

The Intel? Gaudi? 3 AI Accelerator is a high-performance artificial intelligence accelerator launched by Intel, built on the efficient Intel? Gaudi? platform. It boasts outstanding MLPerf benchmark performance and is designed to handle demanding training and inference tasks. The accelerator supports AI applications such as large language models, multimodal models, and enterprise RAG in data centers or the cloud, operating on your existing Ethernet infrastructure. Whether you need a single accelerator or thousands, Intel Gaudi 3 can play a crucial role in your AI success.

This GitHub repository contains training and inference code to replicate our winning solution in the AI Mathematics Olympic (AIMO) Progress Prize 1. Our solution consists of four main parts: a recipe for fine-tuning DeepSeekMath-Base 7B for use in solving math problems using Tool Integrated Reasoning (TIR); two high-quality datasets of about 10 million math questions and solutions; an algorithm for generating solution candidates with coding execution feedback (SC-TIR); and four carefully selected validation sets from AMC, AIME, and MATH to guide model selection and avoid overfitting the public leaderboard.

Gemini 1.5 Flash is the latest AI model released by the Google DeepMind team. It distills core knowledge and skills from the larger 1.5 Pro model through a distillation process, providing a smaller and more efficient model. This model excels in multi-modal reasoning, long text processing, chat applications, image and video captioning, long document and table data extraction. Its significance lies in providing solutions for applications requiring low latency and low-cost services while maintaining high-quality output.

Large language models increasingly rely on distributed techniques for training and inference. These techniques necessitate communication between devices, and as the number of devices increases, this can degrade scaling efficiency. While some distributed techniques can overlap communication to hide independent computation, techniques like tensor parallelism (TP) inherently serialize communication with model execution. One way to hide this serialized communication is to interweave it with producer operations (data generation) in a fine-grained manner. However, implementing this fine-grained communication and computation interleaving in software can be challenging. Furthermore, like any concurrent execution, it requires sharing computational and memory resources between computation and communication, leading to resource contention and decreased overlap efficiency. To overcome these challenges, we propose T3, which uses hardware-software co-design to transparently overlap serialized communication while minimizing resource contention with computation. T3, through simple configuration of producer output address spaces, transparently fuses producer operations and subsequent communication, requiring minimal software changes. At the hardware level, T3 incorporates lightweight tracking and triggering mechanisms to orchestrate producer computation and communication. It further leverages enhanced compute memory for computation related to communication. Consequently, T3 reduces resource contention and effectively overlaps serialized communication with computation. For important Transformer models like T-NLG, T3 achieves a geometric mean speedup of 30% (up to 47%) for communication-intensive sublayers and a geometric mean reduction of 22% (up to 36%) in data movement. Furthermore, T3's benefits persist as models scale: achieving a geometric mean speedup of 29% for sublayers in the 500B parameter sim model, PALM, and MT-NLG.