SCoPE: Evaluating LLMs for Software Vulnerability Detection

José Gonçalves, Tiago Dias, Eva Maia, Isabel Praça
7/19/2024
cs.SEcs.AIcs.CR

No official code found. Best implementation: imahasan/PEFT_experiment_LLM_LoRA_IA3 (1 star, last commit 2025-01-26).

Abstract

In recent years, code security has become increasingly important, especially with the rise of interconnected technologies. Detecting vulnerabilities early in the software development process has demonstrated numerous benefits. Consequently, the scientific community started using machine learning for automated detection of source code vulnerabilities. This work explores and refines the CVEFixes dataset, which is commonly used to train models for code-related tasks, specifically the C/C++ subset. To this purpose, the Source Code Processing Engine (SCoPE), a framework composed of strategized techniques that can be used to reduce the size and normalize C/C++ functions is presented. The output generated by SCoPE was used to create a new version of CVEFixes. This refined dataset was then employed in a feature representation analysis to assess the effectiveness of the tool's code processing techniques, consisting of fine-tuning three pre-trained LLMs for software vulnerability detection. The results show that SCoPE successfully helped to identify 905 duplicates within the evaluated subset. The LLM results corroborate with the literature regarding their suitability for software vulnerability detection, with the best model achieving 53% F1-score.

View on arXivView PDF

Code Implementations(1)

imahasan/PEFT_experiment_LLM_LoRA_IA3Community55%

This repository displays the experiemt of my final thesis work where I am fine tuning CodeGen-2B Multi pre-trained LLM from Hugging Face with a dataset on C and C++ codes curated from CVEFixes (v1.0.7) dataset and further fine tuned by the research paper "Scope: Evaluating llms for software vulnerability detection" and also by this thesis.

10Shell, PythonJan 22, 20251 years ago

Similar Papers

Automated Code-centric Software Vulnerability Assessment: How Far Are We? An Empirical Study in C/C++

88% match1 repo52024

Enhancing Large Language Models with Faster Code Preprocessing for Vulnerability Detection

86% match1 repo12025

Machine Learning Techniques for Python Source Code Vulnerability Detection

86% match1 repo102024

VulnLLMEval: A Framework for Evaluating Large Language Models in Software Vulnerability Detection and Patching

86% match2024

An Empirical Evaluation of LLM-Based Approaches for Code Vulnerability Detection: RAG, SFT, and Dual-Agent Systems

86% match2026

VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models

85% match1 repo382024

Cite this paper

@article{gonalves2024scope,
  title  = {SCoPE: Evaluating LLMs for Software Vulnerability Detection},
  author = {José Gonçalves and Tiago Dias and Eva Maia and Isabel Praça},
  year   = {2024},
  eprint = {2407.14372},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.14372v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.14372.svg)](https://codeofpaper.com/paper/2407.14372)

Discussion