MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation
Abstract
Medical AI has tremendous potential to advance healthcare by supporting the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving provider and patient experience. We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. To meet this need, we are building MedPerf, an open framework for benchmarking machine learning in the medical domain. MedPerf will enable federated evaluation in which models are securely distributed to different facilities for evaluation, thereby empowering healthcare organizations to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status, and our roadmap. We call for researchers and organizations to join us in creating the MedPerf open benchmarking platform.
Cite
@article{arxiv.2110.01406,
title = {MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation},
author = {Alexandros Karargyris and Renato Umeton and Micah J. Sheller and Alejandro Aristizabal and Johnu George and Srini Bala and Daniel J. Beutel and Victor Bittorf and Akshay Chaudhari and Alexander Chowdhury and Cody Coleman and Bala Desinghu and Gregory Diamos and Debo Dutta and Diane Feddema and Grigori Fursin and Junyi Guo and Xinyuan Huang and David Kanter and Satyananda Kashyap and Nicholas Lane and Indranil Mallick and Pietro Mascagni and Virendra Mehta and Vivek Natarajan and Nikola Nikolov and Nicolas Padoy and Gennady Pekhimenko and Vijay Janapa Reddi and G Anthony Reina and Pablo Ribalta and Jacob Rosenthal and Abhishek Singh and Jayaraman J. Thiagarajan and Anna Wuest and Maria Xenochristou and Daguang Xu and Poonam Yadav and Michael Rosenthal and Massimo Loda and Jason M. Johnson and Peter Mattson},
journal= {arXiv preprint arXiv:2110.01406},
year = {2023}
}