DataPerf：面向以数据为中心的人工智能开发的基准套件
机器学习
2023-10-16 v4
摘要
机器学习研究长期聚焦于模型而非数据集，且 prominent 数据集被用于常见 ML 任务时并未考虑底层问题的广度、难度与忠实度。忽视数据的根本重要性已导致现实应用中出现不准确、偏差与脆弱性，且现有数据集基准的饱和也阻碍了研究进展。为此，我们提出 DataPerf，一个由社区主导的用于评估 ML 数据集与以数据为中心的算法的基准套件。我们旨在通过竞争、可比性与可复现性来促进以数据为中心的人工智能创新。我们使 ML 社区能够迭代数据集而非仅迭代架构，并提供开放的在线平台与多轮挑战以支持这种迭代开发。DataPerf 的首次迭代包含五个基准，覆盖视觉、语音、采集、调试与扩散提示中广泛的以数据为中心的技术、任务与模态，且我们支持托管社区贡献的新基准。这些基准、在线评估平台与基线实现均为开源，MLCommons 协会将维护 DataPerf 以确保对学术界与工业界的长期益处。
引用
@article{arxiv.2207.10062,
title = {DataPerf: Benchmarks for Data-Centric AI Development},
author = {Mark Mazumder and Colby Banbury and Xiaozhe Yao and Bojan Karlaš and William Gaviria Rojas and Sudnya Diamos and Greg Diamos and Lynn He and Alicia Parrish and Hannah Rose Kirk and Jessica Quaye and Charvi Rastogi and Douwe Kiela and David Jurado and David Kanter and Rafael Mosquera and Juan Ciro and Lora Aroyo and Bilge Acun and Lingjiao Chen and Mehul Smriti Raje and Max Bartolo and Sabri Eyuboglu and Amirata Ghorbani and Emmett Goodman and Oana Inel and Tariq Kane and Christine R. Kirkpatrick and Tzu-Sheng Kuo and Jonas Mueller and Tristan Thrush and Joaquin Vanschoren and Margaret Warren and Adina Williams and Serena Yeung and Newsha Ardalani and Praveen Paritosh and Lilith Bat-Leah and Ce Zhang and James Zou and Carole-Jean Wu and Cody Coleman and Andrew Ng and Peter Mattson and Vijay Janapa Reddi},
journal= {arXiv preprint arXiv:2207.10062},
year = {2023}
}
备注
NeurIPS 2023 Datasets and Benchmarks Track