PromptBench: A Unified Library for Evaluation of Large Language Models

Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, Xing Xie
12/13/2023
cs.AIcs.CLcs.LG

No official code found. Best implementation: microsoftarchive/promptbench (2,816 stars, MIT, last commit 2026-02-20).

Abstract

The evaluation of large language models (LLMs) is crucial to assess their performance and mitigate potential security risks. In this paper, we introduce PromptBench, a unified library to evaluate LLMs. It consists of several key components that are easily used and extended by researchers: prompt construction, prompt engineering, dataset and model loading, adversarial prompt attack, dynamic evaluation protocols, and analysis tools. PromptBench is designed to be an open, general, and flexible codebase for research purposes that can facilitate original study in creating new benchmarks, deploying downstream applications, and designing new evaluation protocols. The code is available at: https://github.com/microsoft/promptbench and will be continuously supported.

View on arXivView PDF

Code Implementations(1)

microsoftarchive/promptbenchCommunity100%
MIT

A unified evaluation framework for large language models

2,816221PythonJun 13, 20238 months agoMIT
adversarial-attacksbenchmarkchatgptevaluationlarge-language-models+3 more

Similar Papers

State of What Art? A Call for Multi-Prompt LLM Evaluation

91% match1 repo152023

Evaluating Large Language Models: A Comprehensive Survey

88% match1 repo8102023

CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models

88% match2025

Evalverse: Unified and Accessible Library for Large Language Model Evaluation

87% match1 repo2352024

AgentSims: An Open-Source Sandbox for Large Language Model Evaluation

87% match1 repo9602023

OpenCompass: A Universal Evaluation Platform for Large Language Models

87% match2026

Cite this paper

@article{zhu2023promptbench,
  title  = {PromptBench: A Unified Library for Evaluation of Large Language Models},
  author = {Kaijie Zhu and Qinlin Zhao and Hao Chen and Jindong Wang and Xing Xie},
  year   = {2023},
  eprint = {2312.07910},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.07910v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.07910.svg)](https://codeofpaper.com/paper/2312.07910)

Discussion