%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
Openscholar ExpertEval
Overview :
OpenScholar_ExpertEval is a collection of interfaces and scripts for expert evaluation and data assessment, designed to support the OpenScholar project. This project provides detailed human evaluation of model-generated text through retrieval-augmented language model synthesis of scientific literature. The background of this product is based on research projects by AllenAI, offering significant academic and technical value to help researchers and developers better understand and enhance language models.
Target Users :
The target audience includes researchers, developers, and educators, particularly those working in the fields of natural language processing and machine learning. This product is well-suited for them as it provides a platform to evaluate and improve the performance of language models, especially in synthesizing scientific literature.
Use Cases
Researchers use this tool to assess the accuracy and reliability of scientific literature generated by different language models.
Educators can leverage this tool to teach students how to evaluate AI-generated content.
Developers can use this tool to test and improve their own language models.
Features
Provides a manual evaluation annotation interface for experts to assess text generated by models.
Supports RAG evaluation, capable of evaluating retrieval-augmented generation models.
Fine-grained evaluation: Allows experts to conduct more detailed assessments.
Data preparation: Requires evaluation instances in the specified folder, supporting JSONL format.
Results database storage: Evaluation results are stored by default in a local database file.
Results export: Supports exporting evaluation results to Excel files.
Evaluation metric computation: Provides scripts to calculate evaluation metrics and consistency.
Interface sharing: Supports deployment on cloud services for sharing the evaluation interface.
How to Use
1. Set up the environment: Follow the instructions in the README to create and activate a virtual environment, and install the dependencies.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%230080ff;%20}%20%3c/style%3e%3c/defs%3e%3cpath%20class='st0'%20d='M16.2,11.1c.4.5.4,1.2,0,1.8l-4.7,7.1h-3.8l5.3-8L7.6,4h3.8l4.7,7.1Z'/%3e%3c/svg%3e)
2. Prepare data: Place the evaluation instances into the `data` folder, ensuring each instance includes prompts and the completion results of two models.
3. Run the application: Start the evaluation interface using the command `python app.py`.
4. Access the interface: Open `http://localhost:5001` in your browser to access the evaluation interface.
5. Review results: After evaluation is complete, view the progress at `http://localhost:5001/summary`.
6. Export results: Use the command `python export_db.py` to export evaluation results to an Excel file.
7. Calculate metrics: Run the command `python compute_metrics.py` to compute evaluation metrics and consistency.
Featured AI Tools
Tensorpool
TensorPool is a cloud GPU platform dedicated to simplifying machine learning model training. It provides an intuitive command-line interface (CLI) enabling users to easily describe tasks and automate GPU orchestration and execution. Core TensorPool technology includes intelligent Spot instance recovery, instantly resuming jobs interrupted by preemptible instance termination, combining the cost advantages of Spot instances with the reliability of on-demand instances. Furthermore, TensorPool utilizes real-time multi-cloud analysis to select the cheapest GPU options, ensuring users only pay for actual execution time, eliminating costs associated with idle machines. TensorPool aims to accelerate machine learning engineering by eliminating the extensive cloud provider configuration overhead. It offers personal and enterprise plans; personal plans include a $5 weekly credit, while enterprise plans provide enhanced support and features.
Model Training and Deployment
308.6K
English Picks
Ollama
Ollama is a local large language model tool that allows users to quickly run Llama 2, Code Llama, and other models. Users can customize and create their own models. Ollama currently supports macOS and Linux, with a Windows version coming soon. The product aims to provide users with a localized large language model runtime environment to meet their personalized needs.
Model Training and Deployment
275.4K