Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)

Apurv Verma, Satyapriya Krishna, Sebastian Gehrmann, Madhavan Seshadri, Anu Pradhan +5 more
7/20/2024
cs.CLcs.CR

No official code found. Best implementation: dapurv5/awesome-red-teaming-llms (50 stars, CC-BY-4.0, last commit 2026-04-14).

Abstract

Creating secure and resilient applications with large language models (LLM) requires anticipating, adjusting to, and countering unforeseen threats. Red-teaming has emerged as a critical technique for identifying vulnerabilities in real-world LLM implementations. This paper presents a detailed threat model and provides a systematization of knowledge (SoK) of red-teaming attacks on LLMs. We develop a taxonomy of attacks based on the stages of the LLM development and deployment process and extract various insights from previous research. In addition, we compile methods for defense and practical red-teaming strategies for practitioners. By delineating prominent attack motifs and shedding light on various entry points, this paper provides a framework for improving the security and robustness of LLM-based systems.

View on arXivView PDF

Code Implementations(1)

dapurv5/awesome-red-teaming-llmsCommunity100%
CC-BY-4.0

Papers from our SoK on Red-Teaming (Accepted at TMLR)

5013Apr 27, 20246 months agoCC-BY-4.0
adversarial-attacksai-safetyai-securityawesomeawesome-list+3 more

Similar Papers

Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models

91% match2025

RedTWIZ: Diverse LLM Red Teaming via Adaptive Attack Planning

90% match2025

Attack and defense techniques in large language models: A survey and new perspectives

90% match2025

Automated Progressive Red Teaming

89% match1 repo62024

RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models

89% match1 repo292026

Exploring Straightforward Conversational Red-Teaming

88% match2024

Cite this paper

@article{verma2024operationalizing,
  title  = {Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)},
  author = {Apurv Verma and Satyapriya Krishna and Sebastian Gehrmann and Madhavan Seshadri and Anu Pradhan and Tom Ault and Leslie Barrett and David Rabinowitz and John Doucette and NhatHai Phan},
  year   = {2024},
  eprint = {2407.14937},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.14937v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.14937.svg)](https://codeofpaper.com/paper/2407.14937)

Discussion