Embedding-based classifiers can detect prompt injection attacks

Md. Ahsan Ayub, Subhabrata Majumdar
10/29/2024
cs.CRcs.LG

No official code found. Best implementation: AhsanAyub/malicious-prompt-detection (21 stars, last commit 2024-10-30).

Abstract

Large Language Models (LLMs) are seeing significant adoption in every type of organization due to their exceptional generative capabilities. However, LLMs are found to be vulnerable to various adversarial attacks, particularly prompt injection attacks, which trick them into producing harmful or inappropriate content. Adversaries execute such attacks by crafting malicious prompts to deceive the LLMs. In this paper, we propose a novel approach based on embedding-based Machine Learning (ML) classifiers to protect LLM-based applications against this severe threat. We leverage three commonly used embedding models to generate embeddings of malicious and benign prompts and utilize ML classifiers to predict whether an input prompt is malicious. Out of several traditional ML methods, we achieve the best performance with classifiers built using Random Forest and XGBoost. Our classifiers outperform state-of-the-art prompt injection classifiers available in open-source implementations, which use encoder-only neural networks.

View on arXivView PDF

Code Implementations(1)

AhsanAyub/malicious-prompt-detectionCommunity100%

Detection of malicious prompts used to exploit large language models (LLMs) by leveraging supervised machine learning classifiers.

213PythonMay 20, 20241 years ago
llmmachine-learningprompt-injection

Similar Papers

Palisade -- Prompt Injection Detection Framework

88% match2024

Applying Pre-trained Multilingual BERT in Embeddings for Improved Malicious Prompt Injection Attacks Detection

87% match2024

An Early Categorization of Prompt Injection Attacks on Large Language Models

85% match2024

Improved Large Language Model Jailbreak Detection via Pretrained Embeddings

85% match2024

UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Models

85% match2 repos162025

A Systematic Literature Review on LLM Defenses Against Prompt Injection and Jailbreaking: Expanding NIST Taxonomy

85% match2026

Cite this paper

@article{ayub2024embeddingbased,
  title  = {Embedding-based classifiers can detect prompt injection attacks},
  author = {Md. Ahsan Ayub and Subhabrata Majumdar},
  year   = {2024},
  eprint = {2410.22284},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.22284v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.22284.svg)](https://codeofpaper.com/paper/2410.22284)

Discussion