RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models

Quy-Anh Dang, Chris Ngo, Truong-Son Hy
1/7/2026
cs.CL

Official code: knoveleng/redeval (29 stars, MIT, last commit 2026-03-02).

Abstract

As large language models (LLMs) become integral to safety-critical applications, ensuring their robustness against adversarial prompts is paramount. However, existing red teaming datasets suffer from inconsistent risk categorizations, limited domain coverage, and outdated evaluations, hindering systematic vulnerability assessments. To address these challenges, we introduce RedBench, a universal dataset aggregating 37 benchmark datasets from leading conferences and repositories, comprising 29,362 samples across attack and refusal prompts. RedBench employs a standardized taxonomy with 22 risk categories and 19 domains, enabling consistent and comprehensive evaluations of LLM vulnerabilities. We provide a detailed analysis of existing datasets, establish baselines for modern LLMs, and open-source the dataset and evaluation code. Our contributions facilitate robust comparisons, foster future research, and promote the development of secure and reliable LLMs for real-world deployment. Code: https://github.com/knoveleng/redeval

View on arXivView PDF

Code Implementations(1)

knoveleng/redevalOfficial100%
MIT

[ICLR 2026] - Official repo for the paper: "RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models"

294Shell, PythonJul 28, 20257 months agoMIT
ai-alignmentai-safetyevaluationred-teaming

Similar Papers

ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming

89% match1 repo592024

HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

89% match1 repo1.1k2024

Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)

89% match1 repo502024

Automated Progressive Red Teaming

88% match1 repo62024

Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models

88% match2025

Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models

87% match2025

Cite this paper

@article{dang2026redbench,
  title  = {RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models},
  author = {Quy-Anh Dang and Chris Ngo and Truong-Son Hy},
  year   = {2026},
  eprint = {2601.03699},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2601.03699v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2601.03699.svg)](https://codeofpaper.com/paper/2601.03699)

Discussion