Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents

Chaoran Chen, Bingsheng Yao, Ruishi Zou, Wenyue Hua, Weimin Lyu +3 more
2/18/2025
cs.HCcs.CL

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

Role-Playing Agent (RPA) is an increasingly popular type of LLM Agent that simulates human-like behaviors in a variety of tasks. However, evaluating RPAs is challenging due to diverse task requirements and agent designs. This paper proposes an evidence-based, actionable, and generalizable evaluation design guideline for LLM-based RPA by systematically reviewing 1,676 papers published between Jan. 2021 and Dec. 2024. Our analysis identifies six agent attributes, seven task attributes, and seven evaluation metrics from existing literature. Based on these findings, we present an RPA evaluation design guideline to help researchers develop more systematic and consistent evaluation methods.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Role-Playing Evaluation for Large Language Models

87% match1 repo202025

Role-Playing Agents Driven by Large Language Models: Current Status, Challenges, and Future Trends

86% match2026

MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

84% match2026

FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline

83% match2025

Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects

83% match2026

Role-playing Prompt Framework: Generation and Evaluation

83% match2024

Cite this paper

@article{chen2025towards,
  title  = {Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents},
  author = {Chaoran Chen and Bingsheng Yao and Ruishi Zou and Wenyue Hua and Weimin Lyu and Yanfang Ye and Toby Jia-Jun Li and Dakuo Wang},
  year   = {2025},
  eprint = {2502.13012},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2502.13012v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2502.13012.svg)](https://codeofpaper.com/paper/2502.13012)

Discussion