# Pre-training

Crawl4LLM is an open-source web crawling project designed to provide an efficient data crawling solution for the pre-training of Large Language Models (LLMs). It helps researchers and developers obtain high-quality training corpora through intelligent selection and crawling of web data. The tool supports various document scoring methods and allows flexible adjustment of crawling strategies based on configurations to meet different pre-training needs. Developed in Python, the project boasts good scalability and ease of use, making it suitable for both academic research and industrial applications.

Nemotron-CC is a dataset of 6.3 trillion tokens based on Common Crawl. It integrates classifiers, rewrites synthetic data, and reduces reliance on heuristic filters to convert English Common Crawl into a long-term pre-training dataset with 6.3 trillion tokens, 4.4 trillion of which are globally de-duplicated raw tokens, and 1.9 trillion are synthetically generated tokens. This dataset strikes a better balance between accuracy and data volume, making it significant for training large language models.

The Chinese Internet Corpus Resource Platform is a professional website hosted by the China Cybersecurity Association, aiming to provide high-quality and compliant Chinese corpus resources for the pre-training of large AI models. The platform integrates the collaborative strengths of enterprises, universities, and research units, relying on a 'co-build and share' mechanism, forming several high-quality corpora including Chinese Internet Basic Corpus 2.0, People's Daily Mainstream Value Dataset, and National Library Qing and Ming Literature Corpus. These corpora undergo strict data source validation, format cleansing, language filtering, data deduplication, content filtering, and privacy filtering to ensure the legality, authenticity, accuracy, and objectivity of the data. The resources on this platform are of significant importance for promoting national AI technology innovation and industrial development, aiding large models in better understanding and generating Chinese content, and enhancing their knowledge capability and value alignment.

POINTS-Yi-1.5-9B-Chat is a visual language model that incorporates the latest visual language model technologies along with innovations introduced by WeChat AI. The model features significant innovations in pre-training dataset filtering and Model Soup technology, allowing for substantial reductions in dataset size while enhancing model performance. It excels in multiple benchmark tests, marking an important advancement in the field of visual language models.

TableGPT2 is a large multimodal model specifically pre-trained and fine-tuned for tabular data to address the challenges of inadequate integration in practical applications. It was pre-trained and fine-tuned on over 593.8K tables and 2.36M high-quality query-table-output tuples, achieving unprecedented scale. A key innovation of TableGPT2 is its novel table encoder, designed to capture information at both pattern and cell levels, enhancing the model's ability to handle ambiguous queries, missing column names, and irregular tables. On 23 benchmark metrics, the average performance of TableGPT2 improved by 35.20% for the 7B model and by 49.32% for the 72B model, while maintaining robust general language and coding capabilities.

Data-Juicer is a comprehensive multimodal data processing system aimed at delivering higher quality, richer, and more digestible data for large language models (LLMs). It offers a systematic and reusable data processing library, supports collaborative development between data and models, allows rapid iteration through a sandbox lab, and provides features like data and model feedback loops, visualization, and multidimensional automated evaluation, helping users better understand and improve their data and models. Data-Juicer is actively maintained and regularly enhanced with more features, data recipes, and datasets.

The FineWeb dataset contains over 150 billion web pages of cleaned and deduplicated English text sourced from CommonCrawl. Designed specifically for pre-training large language models, it aims to advance the development of open-source models. The dataset has been meticulously processed and filtered to ensure high quality, making it suitable for a variety of natural language processing tasks.

YOLO-World is an advanced real-time open vocabulary object detector based on the You Only Look Once (YOLO) series of detectors. It enhances open vocabulary detection capabilities through visual-language modeling and pre-training on a large dataset. It employs a novel reparameterizable visual-language path aggregation network (RepVL-PAN) and region-text contrastive loss, promoting interaction between visual and linguistic information. YOLO-World efficiently detects a variety of objects in a zero-shot manner, exhibiting high efficiency. On the challenging LVIS dataset, YOLO-World achieves 35.4 AP and 52.0 FPS on a V100, outperforming many state-of-the-art methods in both accuracy and speed. Moreover, fine-tuned YOLO-World demonstrates outstanding performance on multiple downstream tasks, including object detection and open vocabulary instance segmentation.