English

VCAO: Verifier-Centered Agentic Orchestration for Strategic OS Vulnerability Discovery

Computer Science and Game Theory 2026-04-10 v1 Cryptography and Security Operating Systems

Abstract

We formulate operating-system vulnerability discovery as a \emph{repeated Bayesian Stackelberg search game} in which a Large Reasoning Model (LRM) orchestrator allocates analysis budget across kernel files, functions, and attack paths while external verifiers -- static analyzers, fuzzers, and sanitizers -- provide evidence. At each round, the orchestrator selects a target component, an analysis method, and a time budget; observes tool outputs; updates Bayesian beliefs over latent vulnerability states; and re-solves the game to minimize the strategic attacker's expected payoff. We introduce \textsc{VCAO} (\textbf{V}erifier-\textbf{C}entered \textbf{A}gentic \textbf{O}rchestration), a six-layer architecture comprising surface mapping, intra-kernel attack-graph construction, game-theoretic file/function ranking, parallel executor agents, cascaded verification, and a safety governor. Our DOBSS-derived MILP allocates budget optimally across heterogeneous analysis tools under resource constraints, with formal O~(T)\tilde{O}(\sqrt{T}) regret bounds from online Stackelberg learning. Experiments on five Linux kernel subsystems -- replaying 847 historical CVEs and running live discovery on upstream snapshots -- show that \textsc{VCAO} discovers 2.7×2.7\times more validated vulnerabilities per unit budget than coverage-only fuzzing, 1.9×1.9\times more than static-analysis-only baselines, and 1.4×1.4\times more than non-game-theoretic multi-agent pipelines, while reducing false-positive rates reaching human reviewers by 68\%. We release our simulation framework, synthetic attack-graph generator, and evaluation harness as open-source artifacts.

Keywords

Cite

@article{arxiv.2604.08291,
  title  = {VCAO: Verifier-Centered Agentic Orchestration for Strategic OS Vulnerability Discovery},
  author = {Suyash Mishra},
  journal= {arXiv preprint arXiv:2604.08291},
  year   = {2026}
}

Comments

13 Pages

R2 v1 2026-07-01T12:01:15.118Z