Personas with Attitudes: Controlling LLMs for Diverse Data Annotation

Leon Fröhling, Gianluca Demartini, Dennis Assenmacher
10/15/2024
cs.CLcs.HC

No official code found. Best implementation: frohleon/Personas-with-Attitudes (0 stars, last commit 2024-10-16).

Abstract

We present a novel approach for enhancing diversity and control in data annotation tasks by personalizing large language models (LLMs). We investigate the impact of injecting diverse persona descriptions into LLM prompts across two studies, exploring whether personas increase annotation diversity and whether the impacts of individual personas on the resulting annotations are consistent and controllable. Our results show that persona-prompted LLMs produce more diverse annotations than LLMs prompted without personas and that these effects are both controllable and repeatable, making our approach a suitable tool for improving data annotation in subjective NLP tasks like toxicity detection.

View on arXivView PDF

Code Implementations(1)

frohleon/Personas-with-AttitudesCommunity100%

Code repository for a study on the steerability and subjectivity of LLMs for annotation tasks

00Jupyter NotebookOct 15, 20242 years ago

Similar Papers

Quantifying the Persona Effect in LLM Simulations

89% match1 repo22024

Robust Persona-Aware Toxicity Detection with Prompt Optimization and Learned Ensembling

86% match2026

Algorithmic Fairness in NLP: Persona-Infused LLMs for Human-Centric Hate Speech Detection

86% match2025

Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas

85% match2024

Your Language Model Secretly Contains Personality Subnetworks

85% match2026

Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency

85% match2025

Cite this paper

@article{frhling2024personas,
  title  = {Personas with Attitudes: Controlling LLMs for Diverse Data Annotation},
  author = {Leon Fröhling and Gianluca Demartini and Dennis Assenmacher},
  year   = {2024},
  eprint = {2410.11745},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.11745v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.11745.svg)](https://codeofpaper.com/paper/2410.11745)

Discussion