eyeballvul: a future-proof benchmark for vulnerability detection in the wild

Timothee Chauvin
7/11/2024
cs.CRcs.AIcs.LG

No official code found. Best implementation: timothee-chauvin/eyeballvul_experiments (1 star, last commit 2024-09-06).

Abstract

Long contexts of recent LLMs have enabled a new use case: asking models to find security vulnerabilities in entire codebases. To evaluate model performance on this task, we introduce eyeballvul: a benchmark designed to test the vulnerability detection capabilities of language models at scale, that is sourced and updated weekly from the stream of published vulnerabilities in open-source repositories. The benchmark consists of a list of revisions in different repositories, each associated with the list of known vulnerabilities present at that revision. An LLM-based scorer is used to compare the list of possible vulnerabilities returned by a model to the list of known vulnerabilities for each revision. As of July 2024, eyeballvul contains 24,000+ vulnerabilities across 6,000+ revisions and 5,000+ repositories, and is around 55GB in size.

View on arXivView PDF

Code Implementations(1)

timothee-chauvin/eyeballvul_experimentsCommunity100%
12PythonJul 8, 20242 years ago

Similar Papers

Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Study

88% match2024

From Model to Breach: Towards Actionable LLM-Generated Vulnerabilities Reporting

88% match2025

VulnLLMEval: A Framework for Evaluating Large Language Models in Software Vulnerability Detection and Patching

87% match2024

Vulnerability Detection with Code Language Models: How Far Are We?

87% match1 repo2792024

VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models

87% match1 repo382024

From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Security

86% match2024

Cite this paper

@article{chauvin2024eyeballvul,
  title  = {eyeballvul: a future-proof benchmark for vulnerability detection in the wild},
  author = {Timothee Chauvin},
  year   = {2024},
  eprint = {2407.08708},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.08708v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.08708.svg)](https://codeofpaper.com/paper/2407.08708)

Discussion