# Benchmark

M2RAG is a benchmark codebase for retrieval-augmented generation in multimodal contexts. It answers questions by retrieving multimodal documents, evaluating the ability of multimodal large language models (MLLMs) to leverage knowledge from multimodal contexts. The model is evaluated on tasks such as image captioning, multimodal question answering, fact verification, and image re-ranking, aiming to improve the effectiveness of models in multimodal contextual learning. M2RAG provides researchers with a standardized testing platform to help advance the development of multimodal language models.

ZeroBench is a benchmark specifically designed to evaluate the visual understanding capabilities of large multimodal models (LMMs). It challenges the limits of current models through 100 meticulously crafted and rigorously vetted complex questions, along with 334 sub-questions. This benchmark aims to address the shortcomings of existing visual benchmarks by offering a more challenging and high-quality evaluation tool. ZeroBench's primary strengths are its high difficulty, lightweight design, diversity, and high quality, enabling it to effectively differentiate model performance. Additionally, it provides detailed sub-question evaluation, helping researchers better understand the reasoning abilities of the models.

SWE-Lancer, launched by OpenAI, is a benchmark designed to assess the performance of cutting-edge language models in real-world freelance software engineering tasks. This benchmark encompasses a range of independent engineering tasks, from $50 bug fixes to $32,000 feature implementations, as well as managerial tasks such as selecting between technical implementation options. By mapping model performance to monetary value, SWE-Lancer offers a new perspective on researching the economic impact of AI model development and promoting the advancement of related research.

SimpleQA is a factual benchmark test released by OpenAI, designed to measure the ability of language models to answer short, factual questions. By providing a dataset characterized by high accuracy, diversity, and challenge, along with a good researcher experience, it aids in evaluating and enhancing the accuracy and reliability of language models. This benchmark is a significant advancement for training models that can generate factually correct responses, helping to increase their credibility and expand their applications.

LVBench is a benchmark specifically designed for long video understanding, aimed at advancing the ability of multimodal large language models to comprehend videos spanning several hours. This is crucial for real-world applications such as long-term decision-making, in-depth film reviews and discussions, and live sports commentary.

KnowEdit is a knowledge editing benchmark specifically designed for large language models (LLMs). It provides a comprehensive evaluation framework for testing and comparing the effectiveness of different knowledge editing methods in modifying the behavior of LLMs within specific domains, while maintaining overall performance across various inputs. KnowEdit benchmark comprises six distinct datasets, covering various editing types, including fact manipulation, sentiment modification, and hallucination generation. This benchmark aims to assist researchers and developers in better understanding and improving knowledge editing techniques, thereby propelling the continuous development and applications of LLMs.

MMStar is a benchmark dataset designed to assess the multimodal capabilities of large visual language models. It comprises 1500 carefully selected visual language samples, covering 6 core abilities and 18 sub-dimensions. Each sample has undergone human review, ensuring visual dependency, minimizing data leakage, and requiring advanced multimodal capabilities for resolution. In addition to traditional accuracy metrics, MMStar proposes two new metrics to measure data leakage and the practical performance gains of multimodal training. Researchers can use MMStar to evaluate the multimodal capabilities of visual language models across multiple tasks and leverage the new metrics to discover potential issues within models.

PromptBench is a Python package based on PyTorch designed for evaluating Large Language Models (LLM). It offers a user-friendly API for researchers to assess LLMs. Key features include: rapid model performance evaluation, prompting engineering, adversarial prompting assessment, and dynamic evaluation. Its advantages are simplicity of use, allowing for quick assessment of existing datasets and models, as well as easy customization of personal datasets and models. Positioning itself as a unified open-source library for LLM evaluation.

CelebV-Text is a large-scale, high-quality, and diverse face text-video dataset designed to promote research on face text-video generation tasks. The dataset contains 70,000 out-door face video clips, each accompanied by 20 text descriptions covering 40 general appearances, 5 detailed appearances, 6 lighting conditions, 37 actions, 8 emotions and 6 light directions. CelebV-Text has been validated through comprehensive statistical analysis for its superiority in video, text, and text-video correlation, and it constructs a benchmark to standardize the evaluation of face text-video generation tasks.