The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels

Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
5/9/2024
cs.LGcs.CLcs.CY

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

Longstanding data labeling practices in machine learning involve collecting and aggregating labels from multiple annotators. But what should we do when annotators disagree? Though annotator disagreement has long been seen as a problem to minimize, new perspectivist approaches challenge this assumption by treating disagreement as a valuable source of information. In this position paper, we examine practices and assumptions surrounding the causes of disagreement--some challenged by perspectivist approaches, and some that remain to be addressed--as well as practical and normative challenges for work operating under these assumptions. We conclude with recommendations for the data labeling pipeline and avenues for future research engaging with subjectivity and disagreement.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP

86% match2026

Subjective Logic Encodings

85% match2025

Don't Blame the Data, Blame the Model: Understanding Noise and Bias When Learning from Subjective Annotations

83% match1 repo32024

NUTMEG: Separating Signal From Noise in Annotator Disagreement

82% match2025

iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives?

82% match2023

Diverging Preferences: When do Annotators Disagree and do Models Know?

82% match2026

Cite this paper

@article{fleisig2024perspectivist,
  title  = {The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels},
  author = {Eve Fleisig and Su Lin Blodgett and Dan Klein and Zeerak Talat},
  year   = {2024},
  eprint = {2405.05860},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2405.05860v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.05860.svg)](https://codeofpaper.com/paper/2405.05860)

Discussion