Evaluation is a key part of machine learning (ML), yet there is a lack of support and tooling to enable its informed and systematic practice. We introduce Evaluate and Evaluation on the Hub --a set of tools to facilitate the evaluation of models and datasets in ML. Evaluate is a library to support best practices for measurements, metrics, and comparisons of data and models. Its goal is to support reproducibility of evaluation, centralize and document the evaluation process, and broaden evaluation to cover more facets of model performance. It includes over 50 efficient canonical implementations for a variety of domains and scenarios, interactive documentation, and the ability to easily share implementations and outcomes. The library is available at https://github.com/huggingface/evaluate. In addition, we introduce Evaluation on the Hub, a platform that enables the large-scale evaluation of over 75,000 models and 11,000 datasets on the Hugging Face Hub, for free, at the click of a button. Evaluation on the Hub is available at https://huggingface.co/autoevaluate.
@article{arxiv.2210.01970,
title = {Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements},
author = {Leandro von Werra and Lewis Tunstall and Abhishek Thakur and Alexandra Sasha Luccioni and Tristan Thrush and Aleksandra Piktus and Felix Marty and Nazneen Rajani and Victor Mustar and Helen Ngo and Omar Sanseviero and Mario Šaško and Albert Villanova and Quentin Lhoest and Julien Chaumond and Margaret Mitchell and Alexander M. Rush and Thomas Wolf and Douwe Kiela},
journal= {arXiv preprint arXiv:2210.01970},
year = {2022}
}