How Far Have We Gone in Vulnerability Detection Using Large Language Models

Zeyu Gao, Hao Wang, Yuchen Zhou, Wenyu Zhu, Chao Zhang
11/21/2023
cs.AIcs.CLcs.CR

No official code found. Best implementation: Hustcw/VulBench (80 stars, MIT, last commit 2024-11-18).

Abstract

As software becomes increasingly complex and prone to vulnerabilities, automated vulnerability detection is critically important, yet challenging. Given the significant successes of large language models (LLMs) in various tasks, there is growing anticipation of their efficacy in vulnerability detection. However, a quantitative understanding of their potential in vulnerability detection is still missing. To bridge this gap, we introduce a comprehensive vulnerability benchmark VulBench. This benchmark aggregates high-quality data from a wide range of CTF (Capture-the-Flag) challenges and real-world applications, with annotations for each vulnerable function detailing the vulnerability type and its root cause. Through our experiments encompassing 16 LLMs and 6 state-of-the-art (SOTA) deep learning-based models and static analyzers, we find that several LLMs outperform traditional deep learning approaches in vulnerability detection, revealing an untapped potential in LLMs. This work contributes to the understanding and utilization of LLMs for enhanced software security.

View on arXivView PDF

Code Implementations(1)

Hustcw/VulBenchCommunity100%
MIT

This is a benchmark for evaluating the vulnerability discovery ability of automated approaches including Large Language Models (LLMs), deep learning methods and static analyzers

809C, PythonNov 20, 20231 years agoMIT

Similar Papers

Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Study

92% match2024

VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models

92% match1 repo382024

VulnLLMEval: A Framework for Evaluating Large Language Models in Software Vulnerability Detection and Patching

90% match2024

LProtector: An LLM-driven Vulnerability Detection System

90% match2024

LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights

90% match2025

DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection

89% match1 repo1902023

Cite this paper

@article{gao2023have,
  title  = {How Far Have We Gone in Vulnerability Detection Using Large Language Models},
  author = {Zeyu Gao and Hao Wang and Yuchen Zhou and Wenyu Zhu and Chao Zhang},
  year   = {2023},
  eprint = {2311.12420},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.12420v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.12420.svg)](https://codeofpaper.com/paper/2311.12420)

Discussion