HonestCyberEval: An AI Cyber Risk Benchmark for Automated Software Exploitation

Dan Ristea, Vasilios Mavroudis
10/29/2024
cs.CRcs.AI

No confident code implementation found for this paper.

Abstract

We introduce HonestCyberEval, a new benchmark for assessing AI models' capabilities and risks in automated software exploitation, focusing on their ability to detect and exploit vulnerabilities in real-world software systems. Our evaluation leverages the Nginx web server repository augmented with synthetic vulnerabilities. We assess several leading language models, including OpenAI's GPT-4.5, o3-mini, o1 and o1-mini, Anthropic's Claude-3-7-sonnet-20250219, Claude-3.5-sonnet-20241022 and Claude-3.5-sonnet-20240620, Google DeepMind's Gemini-1.5-pro, and OpenAI's earlier GPT-4o model. Our findings reveal that these models vary significantly in their success rates and efficiency, with o1-preview achieving the highest success rate (92.85\%) and o3-mini and Claude-3.7-sonnet-20250219 providing cost-effective but less successful alternatives. This risk evaluation establishes a foundation for systematically evaluating the AI cyber risk in realistic cyber offence operations.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Towards Automated Penetration Testing: Introducing LLM Benchmark, Analysis, and Improvements

88% match1 repo232024

Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security

88% match2025

ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense

88% match2026

CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities

88% match1 repo2822026

SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI

88% match2024

VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models

87% match1 repo382024

Cite this paper

@article{ristea2024honestcybereval,
  title  = {HonestCyberEval: An AI Cyber Risk Benchmark for Automated Software Exploitation},
  author = {Dan Ristea and Vasilios Mavroudis},
  year   = {2024},
  eprint = {2410.21939},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.21939v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.21939.svg)](https://codeofpaper.com/paper/2410.21939)

Discussion