English

Deployment-complete benchmarking

Machine Learning 2026-05-26 v1 Machine Learning

Abstract

Benchmarks increasingly guide deployment, procurement and scientific screening, yet a score supports only the response it records, not necessarily the deployment action. We introduce deployment-complete benchmarking, which tests whether benchmark evidence determines a deployment action. A benchmark is complete for a claim exactly when the action is constant on each evidence fiber; mixed fibers expose missing deployment information, and completion curves quantify the evidence required to resolve ambiguity. In controlled response spaces, benchmark-channel conformal coverage of 94.98% transferred poorly to an unmeasured deployment channel (10.07%), whereas response-rank intervals achieved 94.91% coverage; even zero benchmark error certified only 45.4% of candidates at the largest residual size. Public audits revealed incompleteness, including 97.9% mixed Tox21 fibers and zero median certifiable fraction in main Matbench and JARVIS audits. In held-out replays, certify-then-acquire reduced false decisions from 1.19% to 0.027% in Tox21 and from 20.3% to 0.128% in JARVIS, while changing model choice and identifying deployment-relevant probes. Deployment-ready benchmarks should report evidence, supported actions, ambiguity and completion cost rather than scores alone.

Keywords

Cite

@article{arxiv.2605.25997,
  title  = {Deployment-complete benchmarking},
  author = {El Mustapha Mansouri and Keigo Arai},
  journal= {arXiv preprint arXiv:2605.25997},
  year   = {2026}
}

Comments

33 pages, 5 figures, 1 table; supplementary tables and code available

R2 v1 2026-07-22T07:32:47.848Z