There is a tendency across different subfields in AI to valorize a small collection of influential benchmarks. These benchmarks operate as stand-ins for a range of anointed common problems that are frequently framed as foundational milestones on the path towards flexible and generalizable AI systems. State-of-the-art performance on these benchmarks is widely understood as indicative of progress towards these long-term goals. In this position paper, we explore the limits of such benchmarks in order to reveal the construct validity issues in their framing as the functionally "general" broad measures of progress they are set up to be.
@article{arxiv.2111.15366,
title = {AI and the Everything in the Whole Wide World Benchmark},
author = {Inioluwa Deborah Raji and Emily M. Bender and Amandalynne Paullada and Emily Denton and Alex Hanna},
journal= {arXiv preprint arXiv:2111.15366},
year = {2021}
}
Comments
Accepted in NeurIPS 2021 Benchmarks and Datasets track