English

VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean

Software Engineering 2026-02-23 v1 Computation and Language Machine Learning Programming Languages

Abstract

Large language models have achieved striking results in interactive theorem proving, particularly in Lean. However, most benchmarks for LLM-based proof automation are drawn from mathematics in the Mathlib ecosystem, whereas proofs in software verification are developed inside definition-rich codebases with substantial project-specific libraries. We introduce VeriSoftBench, a benchmark of 500 Lean 4 proof obligations drawn from open-source formal-methods developments and packaged to preserve realistic repository context and cross-file dependencies. Our evaluation of frontier LLMs and specialized provers yields three observations. First, provers tuned for Mathlib-style mathematics transfer poorly to this repository-centric setting. Second, success is strongly correlated with transitive repository dependence: tasks whose proofs draw on large, multi-hop dependency closures are less likely to be solved. Third, providing curated context restricted to a proof's dependency closure improves performance relative to exposing the full repository, but nevertheless leaves substantial room for improvement. Our benchmark and evaluation suite are released at https://github.com/utopia-group/VeriSoftBench.

Keywords

Cite

@article{arxiv.2602.18307,
  title  = {VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean},
  author = {Yutong Xin and Qiaochu Chen and Greg Durrett and Işil Dillig},
  journal= {arXiv preprint arXiv:2602.18307},
  year   = {2026}
}
R2 v1 2026-07-01T10:44:21.596Z