SLM_Survey is a dedicated research project focused on small language models (SLMs), aiming to provide an in-depth understanding and technical assessment of these models through research and measurement. The project covers transformer-based, decoder-only language models with parameter sizes ranging from 100M to 5B. By investigating 59 state-of-the-art open-source SLMs, it analyzes their technological innovations and evaluates their capabilities across multiple domains, including common-sense reasoning, context learning, mathematics, and programming. Additionally, runtime costs such as inference latency and memory usage have been benchmarked. This research is of significant value in advancing the study of SLMs.

Cheating LLM Benchmarks is a research initiative aimed at exploring cheating behaviors in automated language model (LLM) benchmarking by constructing what are known as 'null models.' The project’s experiments have revealed that even simple null models can achieve high win rates in these benchmarks, challenging the validity and reliability of current benchmarking practices. This research is crucial for understanding the limitations of current language models and improving benchmarking methodologies.

Paper Central is a comprehensive and convenient academic platform launched by Hugging Face. It consolidates open-source academic resources from various sources such as arXiv, Hugging Face paper pages, models, datasets, Spaces, GitHub, and conference proceedings, helping researchers and developers quickly access and utilize open-source resources.

SciAgentsDiscovery is a system that automates scientific research using multi-agent systems and large-scale ontological knowledge graphs. By integrating large language models, data retrieval tools, and multi-agent learning systems, it autonomously generates and refines research hypotheses, revealing potential mechanisms, design principles, and unexpected material properties. The system demonstrates its ability to discover interdisciplinary relationships, particularly in the field of bio-inspired materials, surpassing traditional human-driven research methods.

AI-Researcher is based on a research project by Stanford University's Natural Language Processing team, designed to assist in the generation and management of research ideas through artificial intelligence technology. This tool takes natural language input of research topics and outputs a series of project proposals, ranking and filtering them to help researchers quickly find innovative and feasible concepts. It includes modules for relevant paper searches, retrieval-based idea generation, idea deduplication, project proposal generation, proposal ranking, and filtering.

OpenResearcher is an advanced scientific research assistant designed to provide helpful answers for research inquiries. It can access the arXiv corpus, offering the latest scientific insights to help users explore the forefront of science.

Turtle Benchmark is a new, cheat-proof benchmark based on the 'Turtle Soup' game, focusing on the assessment of large language models (LLMs) in terms of logical reasoning and context comprehension. By eliminating the need for background knowledge, it provides objective and unbiased test results with quantifiable outcomes, ensuring that models cannot be 'gamed' through the use of real user-generated questions.

prism-alignment, created by HannahRoseKirk, is a dataset focused on studying the preferences and value alignment of large language models (LLMs). The dataset collects ratings and feedback from participants of diverse nationalities and cultural backgrounds on model responses gathered through surveys and multi-turn conversations with language models. This data is crucial for understanding and improving the value alignment of artificial intelligence.

Open LLM Leaderboard, hosted by Hugging Face, is a space designed to showcase and compare the performance of various large language models. It offers a platform for developers, researchers, and enterprises to view the performance of different models on specific tasks, ultimately helping users choose the model best suited to their needs.

SLM_Survey is a dedicated research project focused on small language models (SLMs), aiming to provide an in-depth understanding and technical assessment of these models through research and measurement. The project covers transformer-based, decoder-only language models with parameter sizes ranging from 100M to 5B. By investigating 59 state-of-the-art open-source SLMs, it analyzes their technological innovations and evaluates their capabilities across multiple domains, including common-sense reasoning, context learning, mathematics, and programming. Additionally, runtime costs such as inference latency and memory usage have been benchmarked. This research is of significant value in advancing the study of SLMs.

The State of AI Report 2024 is an annual report produced by AI investor Nathan Benaich and Air Street Capital, aimed at analyzing the most interesting developments in the AI field and sparking informed discussions about the current state of AI and its future impacts. The report covers various dimensions including research, industry, politics, and security, providing predictions and assessments for the future of AI development. This report is an essential resource for understanding the latest advancements and trends in AI, making it highly valuable to researchers, entrepreneurs, policymakers, and the interested public.

Cheating LLM Benchmarks is a research initiative aimed at exploring cheating behaviors in automated language model (LLM) benchmarking by constructing what are known as 'null models.' The project’s experiments have revealed that even simple null models can achieve high win rates in these benchmarks, challenging the validity and reliability of current benchmarking practices. This research is crucial for understanding the limitations of current language models and improving benchmarking methodologies.

MLE-bench is a benchmark test launched by OpenAI to measure the performance of AI agents in the domain of machine learning engineering. It compiles 75 diverse challenges from Kaggle-related machine learning engineering competitions, testing real-world skills such as model training, dataset preparation, and experiment execution. Using publicly available leaderboard data from Kaggle, human benchmarks for each competition are established. Various cutting-edge language models are evaluated against this benchmark using open-source agent frameworks, revealing that the best-performing setup—OpenAI's o1-preview paired with the AIDE framework—achieved at least Kaggle bronze medal levels in 16.9% of the competitions. Moreover, the study examines various resource extension forms of AI agents and the effects of pre-training contamination. The benchmark code for MLE-bench has been open-sourced to facilitate future understanding of AI agents' capabilities in machine learning engineering.

Paper Central is a comprehensive and convenient academic platform launched by Hugging Face. It consolidates open-source academic resources from various sources such as arXiv, Hugging Face paper pages, models, datasets, Spaces, GitHub, and conference proceedings, helping researchers and developers quickly access and utilize open-source resources.

SciAgentsDiscovery is a system that automates scientific research using multi-agent systems and large-scale ontological knowledge graphs. By integrating large language models, data retrieval tools, and multi-agent learning systems, it autonomously generates and refines research hypotheses, revealing potential mechanisms, design principles, and unexpected material properties. The system demonstrates its ability to discover interdisciplinary relationships, particularly in the field of bio-inspired materials, surpassing traditional human-driven research methods.

AI-Researcher is based on a research project by Stanford University's Natural Language Processing team, designed to assist in the generation and management of research ideas through artificial intelligence technology. This tool takes natural language input of research topics and outputs a series of project proposals, ranking and filtering them to help researchers quickly find innovative and feasible concepts. It includes modules for relevant paper searches, retrieval-based idea generation, idea deduplication, project proposal generation, proposal ranking, and filtering.

OpenResearcher is an advanced scientific research assistant designed to provide helpful answers for research inquiries. It can access the arXiv corpus, offering the latest scientific insights to help users explore the forefront of science.

SWE-bench Verified is a subset of SWE-bench released by OpenAI that has been manually verified to reliably assess the ability of AI models to solve real-world software issues. It challenges AI to generate patches that resolve the described problems by providing code repositories and problem descriptions. This tool has been developed to improve the accuracy of evaluating the model's ability to autonomously perform software engineering tasks and is a key component of OpenAI's medium-risk framework.