# Pre-trained model

Aria-Base-64K is one of the foundational models in the Aria series, designed for research purposes and further training. It emerged after the long text pre-training phase, trained on 33 billion tokens (21 billion multimodal and 12 billion language tokens, with 69% being long texts). It is suitable for further pre-training or fine-tuning on long video question answering datasets or long document question answering datasets, even in resource-constrained environments, through post-training with short instruction tuning datasets tailored for long text scenarios. The model can comprehend up to 250 high-resolution images or up to 500 medium-resolution images, maintaining robust foundational performance in both language and multimodal contexts.

GLM-4-9B-Chat is the open-source version of the latest pre-trained model GLM-4 series released by Zhi谱 AI. It boasts advanced features such as multi-turn dialogue, web browsing, code execution, custom tool invocation, and long-text reasoning. It supports 26 languages including Japanese, Korean, and German, and has launched a model with support for a 1M context length.

CogVLM2, developed by a team from Tsinghua University, is a second-generation multimodal pre-trained dialogue model. It has achieved significant improvements in multiple benchmark tests, supports 8K content length and a resolution of 1344*1344 for images. CogVLM2 offers both Chinese and English versions that are open-source, achieving performance comparable to some non-open-source models.

Qwen1.5 is a decoder language model series based on the Transformer architecture, including models of various sizes. It features SwiGLU activation, attention QKV bias, and group query attention. It supports multiple natural languages and code. Fine-tuning is recommended, such as SFT, RLHF, etc. Pricing is free.

SpacTor is a new training procedure that includes (1) a mixed objective combining span corruption (SC) and replacement tag detection (RTD), and (2) a two-stage curriculum that optimizes the mixed objective in the initial \tau iterations and then transitions to standard SC loss. Experiments on various NLP tasks, using the encoder-decoder architecture (T5), show that SpacTor-T5 achieves comparable downstream performance to standard SC pre-training while reducing the pre-training iterations by 50% and the total FLOPs by 40%. Additionally, under the same computational budget, we find that SpacTor can significantly improve downstream benchmark performance.

Yi is a globally leading pre-trained model that has achieved SOTA international best performance indicators in multiple evaluations. It has a lightweight model size, surpassing large-scale open-source models, making it more friendly to the developer community. Suitable for individuals and research purposes, Yi boasts the ability to emerge as a large model, catering to diverse scenarios and satisfying the rigid needs of the open-source community. Yi's open-source model is completely open to academic research and simultaneously offers free commercial use application.