Role-Playing Evaluation for Large Language Models

Yassine El Boudouri, Walter Nuninger, Julian Alvarez, Yvan Peter
5/19/2025
cs.CLcs.AI

Official code: yelboudouri/RPEval (20 stars, last commit 2026-03-22).

Abstract

Large Language Models (LLMs) demonstrate a notable capacity for adopting personas and engaging in role-playing. However, evaluating this ability presents significant challenges, as human assessments are resource-intensive and automated evaluations can be biased. To address this, we introduce Role-Playing Eval (RPEval), a novel benchmark designed to assess LLM role-playing capabilities across four key dimensions: emotional understanding, decision-making, moral alignment, and in-character consistency. This article details the construction of RPEval and presents baseline evaluations. Our code and dataset are available at https://github.com/yelboudouri/RPEval

View on arXivView PDF

Code Implementations(1)

yelboudouri/RPEvalOfficial100%

Role-Playing Evaluation for Large Language Models

202PythonNov 8, 20247 months ago
evaluationllms

Similar Papers

RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing

89% match2025

On the Decision-Making Abilities in Role-Playing using Large Language Models

89% match2024

Role-playing Prompt Framework: Generation and Evaluation

89% match2024

Role-Playing Agents Driven by Large Language Models: Current Status, Challenges, and Future Trends

89% match2026

Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents

87% match2025

The Oscars of AI Theater: A Survey on Role-Playing with Language Models

86% match1 repo2292024

Cite this paper

@article{boudouri2025roleplaying,
  title  = {Role-Playing Evaluation for Large Language Models},
  author = {Yassine El Boudouri and Walter Nuninger and Julian Alvarez and Yvan Peter},
  year   = {2025},
  eprint = {2505.13157},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.13157v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.13157.svg)](https://codeofpaper.com/paper/2505.13157)

Discussion