# Language model

OuteTTS-0.1-350M is a text-to-speech synthesis technology based on a pure language model, requiring no external adapters or complex architectures, achieving high-quality voice synthesis through carefully designed prompts and audio tokenization. This model is based on the LLaMa architecture, utilizing 350 million parameters to demonstrate the potential for direct voice synthesis using language models. It processes audio in three steps: using WavTokenizer for audio tokenization, creating precise word-to-audio mappings through CTC forced alignment, and generating structured prompts that follow specific formats. The key advantages of OuteTTS include a pure language modeling approach, voice cloning capabilities, and compatibility with llama.cpp and GGUF formats.

AMD-Llama-135m is a language model trained on the LLaMA2 architecture, capable of smooth loading and usage on the AMD MI250 GPU. This model supports both text and code generation and is applicable to various natural language processing tasks.

DCLM-baseline is a pretraining dataset for language model benchmarking, containing 4T tokens and 3B documents. It is curated from the Common Crawl dataset after a careful planning of data cleaning, filtering, and deduplication steps, aiming to demonstrate the importance of data curation in training efficient language models. The dataset is only for research purposes and should not be used in production environments or for training domain-specific models, such as those for code and mathematics.

Qwen2-Audio is a large audio language model proposed by Alibaba Cloud, capable of processing various audio signals as input and performing audio analysis or direct text reply based on speech commands. The model supports two different audio interaction modes: voice chat and audio analysis. It has achieved outstanding performance in 13 standard benchmark tests, including automatic speech recognition, speech-to-text translation, and speech emotion recognition.

H2O-Danube2-1.8B is the latest open-source tiny language model from H2O.ai, designed specifically for offline and enterprise applications. It features an economically efficient interface and training cost, making it easy to integrate into edge devices like smartphones and drones. This model ranks first in the <2B range on the Hugging Face Open LLM Leaderboard, offering up to 200 times cost savings on queries and better accuracy in document processing, with cost reductions up to 100%. The H2O.ai platform also provides cost control and flexibility, supporting the mixed use of more than 30 Large Language Models (LLMs), including proprietary and open-source LLMs.

Gemma-2-9B-Chinese-Chat is an instruction-tuned language model based on google/gemma-2-9b-it, specifically designed for Chinese and English users. It boasts capabilities such as role-playing and tool usage. Fine-tuned through the ORPO algorithm, the model significantly enhances the accuracy of responses to Chinese queries, minimizes issues with mixed Chinese and English usage, and excels in role-playing, tool usage, and mathematical calculations.

Index-1.9B-Character is a large language model independently developed by the Index team, focusing on the field of role-playing. It has a parameter scale of 1.9 billion. The model supports users to quickly customize roles through uploading roleplay dialogue data, and has high role consistency, dialogue ability, and role-playing attractiveness. In the authoritative benchmark CharacterEval evaluation, it ranked ninth overall, outperforming models of the same scale.

Prometheus-Eval is an open-source toolkit designed to assess the performance of large language models (LLMs) in generation tasks. It provides a straightforward interface for evaluating instructions and responses using the Prometheus model. The Prometheus 2 model supports direct evaluation (absolute scoring) and paired ranking (relative scoring), which can simulate human judgment and proprietary language model-based evaluation, addressing issues of fairness, control, and affordability.

RULER is a new synthetic benchmark that provides a more comprehensive evaluation of long-text language models. It extends standard retrieval tests to cover different types and quantities of information points. Additionally, RULER introduces new task categories, such as multi-hop tracking and aggregation, to test behaviors beyond retrieving from context. 10 long-text language models were evaluated on RULER and achieved performance on 13 representative tasks. Despite achieving near-perfect accuracy on standard retrieval tests, these models performed poorly as context length increased. Only four models (GPT-4, Command-R, Yi-34B, and Mixtral) performed reasonably well at a length of 32K. We make RULER publicly available to promote comprehensive evaluation of long-text language models.

Jamba is an open-weights language model based on the hybrid SSM-Transformer architecture, delivering top-tier quality and performance. It combines the strengths of Transformer and SSM architectures, achieving outstanding results in inference benchmarks while providing a 3x throughput increase in long-context scenarios. Jamba is currently the only model of this scale that can support a 140,000-character context on a single GPU, offering exceptional cost-effectiveness. As a foundational model, Jamba is designed for developers to fine-tune, train, and build customized solutions.

Baichuan 3, a large language model with over trillion parameters developed by Baichuan Intelligent, has demonstrated outstanding performance in multiple authoritative general ability assessments, particularly exceeding GPT-4 in Chinese tasks. It excels in natural language processing, code generation, and medical tasks. It employs several innovative techniques to enhance model capabilities, including dynamic data selection, importance preservation, and asynchronous Checkpoint storage. The training process utilizes a dynamic data selection scheme based on causal sampling to ensure data quality. An importance preservation progressive initialization method is introduced to optimize model training stability. A series of optimizations have also been implemented for parallel training, resulting in a performance improvement of over 30%.

Bard Advanced is a language model service expected to be launched by Google, built on the more powerful Gemini Ultra model. Users need to subscribe to Google One to access Bard Advanced. Compared to the free version of Bard, Bard Advanced has more advanced mathematical and reasoning skills, can answer user questions of higher quality, and supports the creation of custom conversational robots. Bard Advanced provides users with a more intelligent and professional language generation service.

Flash-Decoding is a technique for long-context inference that can significantly accelerate the attention mechanism during inference, leading to an 8x improvement in generation speed. This technique achieves faster inference speed by parallelly loading keys and values and then rescaling and combining the results to maintain the correct attention output. Flash-Decoding is suitable for large language models and can handle long contexts such as long documents, long conversations, or entire codebases. Flash-Decoding is available in the FlashAttention package and xFormers, which can automatically select between Flash-Decoding and FlashAttention methods. It can also utilize the efficient Triton kernel.