Long contexts of recent LLMs have enabled a new use case: asking models to find security vulnerabilities in entire codebases. To evaluate model performance on this task, we introduce eyeballvul: a benchmark designed to test the vulnerability detection capabilities of language models at scale, that is sourced and updated weekly from the stream of published vulnerabilities in open-source repositories. The benchmark consists of a list of revisions in different repositories, each associated with the list of known vulnerabilities present at that revision. An LLM-based scorer is used to compare the list of possible vulnerabilities returned by a model to the list of known vulnerabilities for each revision. As of July 2024, eyeballvul contains 24,000+ vulnerabilities across 6,000+ revisions and 5,000+ repositories, and is around 55GB in size.
@article{arxiv.2407.08708,
title = {eyeballvul: a future-proof benchmark for vulnerability detection in the wild},
author = {Timothee Chauvin},
journal= {arXiv preprint arXiv:2407.08708},
year = {2024}
}
Comments
Due to a bug in the litellm library (we haven't tracked exactly which one, but probably at least https://github.com/BerriAI/litellm/commit/2452753e084e8134c0c484b32c63fb5f2950c5ba), our Gemini 1.5 Pro inference costs were incorrect. We've updated the relevant plot (Fig 7) and its interpretation (both Gemini 1.5 Pro and Claude 3.5 Sonnet stand out from the other models, not just Gemini 1.5 Pro)