%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
Hallucination Leaderboard
Overview :
This product is an open-source project developed by Vectara to evaluate the hallucination rate of Large Language Models (LLMs) when summarizing short documents. It utilizes Vectara's Hughes Hallucination Evaluation Model (HHEM-2.1) to calculate rankings by detecting hallucinations in the model's output. This tool is significant for researching and developing more reliable LLMs, helping developers understand and improve the accuracy of their models.
Target Users :
This product is primarily aimed at researchers, developers, and users interested in evaluating LLM performance in the field of Natural Language Processing (NLP). It helps them understand the accuracy and reliability of different LLMs when generating content, enabling them to choose the most suitable model for specific tasks.
Use Cases
Researchers can use this leaderboard to compare the hallucination rates of different LLM models when generating summaries, thereby selecting more reliable models.
Developers can use this tool to evaluate the performance of their own LLM models and optimize them to reduce hallucination.
Businesses can refer to this leaderboard to select LLM models suitable for their business needs, for applications such as content generation and customer service.
Features
Provides LLM hallucination evaluation based on the HHEM-2.1 model.
Supports comparison and ranking of various LLMs.
Uses the CNN/Daily Mail corpus for document summarization testing.
Evaluates models through API calls to various LLM models.
Provides key metrics such as hallucination rate, factuality rate, and answer rate.
Supports the evaluation of multilingual models (currently primarily supports English).
Regularly updated to reflect changes in model performance.
How to Use
1. Visit the project homepage (https://github.com/vectara/hallucination-leaderboard) to understand the project background and usage instructions.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%230080ff;%20}%20%3c/style%3e%3c/defs%3e%3cpath%20class='st0'%20d='M16.2,11.1c.4.5.4,1.2,0,1.8l-4.7,7.1h-3.8l5.3-8L7.6,4h3.8l4.7,7.1Z'/%3e%3c/svg%3e)
2. Refer to the README file to learn how to use the HHEM-2.1 model for evaluation.
3. Prepare the LLM models to be evaluated and their corresponding API interfaces.
4. Use the project's provided scripts or code to call the LLM models and generate summaries.
5. Evaluate the generated summaries using the HHEM-2.1 model to obtain metrics such as hallucination rate.
6. Analyze the evaluation results and compare the performance of different models.
7. Adjust the models or select a better model for applications based on your needs.
Featured AI Tools
Tensorpool
TensorPool is a cloud GPU platform dedicated to simplifying machine learning model training. It provides an intuitive command-line interface (CLI) enabling users to easily describe tasks and automate GPU orchestration and execution. Core TensorPool technology includes intelligent Spot instance recovery, instantly resuming jobs interrupted by preemptible instance termination, combining the cost advantages of Spot instances with the reliability of on-demand instances. Furthermore, TensorPool utilizes real-time multi-cloud analysis to select the cheapest GPU options, ensuring users only pay for actual execution time, eliminating costs associated with idle machines. TensorPool aims to accelerate machine learning engineering by eliminating the extensive cloud provider configuration overhead. It offers personal and enterprise plans; personal plans include a $5 weekly credit, while enterprise plans provide enhanced support and features.
Model Training and Deployment
306.9K
English Picks
Ollama
Ollama is a local large language model tool that allows users to quickly run Llama 2, Code Llama, and other models. Users can customize and create their own models. Ollama currently supports macOS and Linux, with a Windows version coming soon. The product aims to provide users with a localized large language model runtime environment to meet their personalized needs.
Model Training and Deployment
263.0K