CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models

Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich +8 more
8/2/2024
cs.CRcs.LG

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

We are releasing a new suite of security benchmarks for LLMs, CYBERSECEVAL 3, to continue the conversation on empirically measuring LLM cybersecurity risks and capabilities. CYBERSECEVAL 3 assesses 8 different risks across two broad categories: risk to third parties, and risk to application developers and end users. Compared to previous work, we add new areas focused on offensive security capabilities: automated social engineering, scaling manual offensive cyber operations, and autonomous offensive cyber operations. In this paper we discuss applying these benchmarks to the Llama 3 models and a suite of contemporaneous state-of-the-art LLMs, enabling us to contextualize risks both with and without mitigations in place.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

LLM Cyber Evaluations Don't Capture Real-World Risk

91% match1 repo52025

Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques

89% match2025

CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models

89% match1 repo4.3k2024

Large Language Models in Cybersecurity: State-of-the-Art

87% match2024

Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposal

86% match2024

A Survey of Large Language Models in Cybersecurity

85% match2024

Cite this paper

@article{wan2024cyberseceval,
  title  = {CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models},
  author = {Shengye Wan and Cyrus Nikolaidis and Daniel Song and David Molnar and James Crnkovich and Jayson Grace and Manish Bhatt and Sahana Chennabasappa and Spencer Whitman and Stephanie Ding and Vlad Ionescu and Yue Li and Joshua Saxe},
  year   = {2024},
  eprint = {2408.01605},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.01605v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.01605.svg)](https://codeofpaper.com/paper/2408.01605)

Discussion