English

SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks

Machine Learning 2026-05-26 v1

Abstract

Pairwise model comparisons drawn from foundation-model benchmarks ("A is safer than B") are read as quantitative verdicts but hinge on harness choices benchmark papers under-specify. We close one theory-benchmark loop on this primitive: a finite-envelope proposition tying a measurable pairwise-disagreement rate to whether the strict ordering admits a configuration-pair reversal, paired with a commit-stamped evaluation protocol that operationalises it on widely cited alignment benchmarks. On every benchmark we test, configuration choice alone can flip the pairwise verdict; the proposition isolates this strict-reversal failure mode.

Keywords

Cite

@article{arxiv.2605.25492,
  title  = {SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks},
  author = {Yanhang Li and Zhichao Fan and Zexin Zhuang},
  journal= {arXiv preprint arXiv:2605.25492},
  year   = {2026}
}