NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
Abstract
We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.
Cite
@article{arxiv.2101.00133,
title = {NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned},
author = {Sewon Min and Jordan Boyd-Graber and Chris Alberti and Danqi Chen and Eunsol Choi and Michael Collins and Kelvin Guu and Hannaneh Hajishirzi and Kenton Lee and Jennimaria Palomaki and Colin Raffel and Adam Roberts and Tom Kwiatkowski and Patrick Lewis and Yuxiang Wu and Heinrich Küttler and Linqing Liu and Pasquale Minervini and Pontus Stenetorp and Sebastian Riedel and Sohee Yang and Minjoon Seo and Gautier Izacard and Fabio Petroni and Lucas Hosseini and Nicola De Cao and Edouard Grave and Ikuya Yamada and Sonse Shimaoka and Masatoshi Suzuki and Shumpei Miyawaki and Shun Sato and Ryo Takahashi and Jun Suzuki and Martin Fajcik and Martin Docekal and Karel Ondrej and Pavel Smrz and Hao Cheng and Yelong Shen and Xiaodong Liu and Pengcheng He and Weizhu Chen and Jianfeng Gao and Barlas Oguz and Xilun Chen and Vladimir Karpukhin and Stan Peshterliev and Dmytro Okhonko and Michael Schlichtkrull and Sonal Gupta and Yashar Mehdad and Wen-tau Yih},
journal= {arXiv preprint arXiv:2101.00133},
year = {2021}
}
Comments
26 pages; Published in Proceedings of Machine Learning Research (PMLR), NeurIPS 2020 Competition and Demonstration Track