Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard
密码学与安全
2026-05-22 v1 人工智能
摘要
The benchmarks used to evaluate AI agents in security-critical roles suffer from crucial weaknesses. Building on recent empirical evidence, we characterize three core challenges that undermine security evaluations: benchmark vulnerabilities, temporal staleness, and runtime uncertainty. We then outline practical directions toward building more robust and trustworthy evaluation frameworks.
引用
@article{arxiv.2605.22568,
title = {Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard},
author = {Sahar Abdelnabi and Chris Hicks and Konrad Rieck and Ahmad-Reza Sadeghi},
journal= {arXiv preprint arXiv:2605.22568},
year = {2026}
}