Rater Cohesion and Quality from a Vicarious Perspective

Deepak Pandita, Tharindu Cyril Weerasooriya, Sujan Dutta, Sarah K. Luger, Tharindu Ranasinghe +3 more
8/15/2024
cs.CL

No official code found. Best implementation: Homan-Lab/rater_cohesion_public (0 stars, Apache-2.0, last commit 2026-03-19).

Abstract

Human feedback is essential for building human-centered AI systems across domains where disagreement is prevalent, such as AI safety, content moderation, or sentiment analysis. Many disagreements, particularly in politically charged settings, arise because raters have opposing values or beliefs. Vicarious annotation is a method for breaking down disagreement by asking raters how they think others would annotate the data. In this paper, we explore the use of vicarious annotation with analytical methods for moderating rater disagreement. We employ rater cohesion metrics to study the potential influence of political affiliations and demographic backgrounds on raters' perceptions of offense. Additionally, we utilize CrowdTruth's rater quality metrics, which consider the demographics of the raters, to score the raters and their annotations. We study how the rater quality metrics influence the in-group and cross-group rater cohesion across the personal and vicarious levels.

View on arXivView PDF

Code Implementations(1)

Homan-Lab/rater_cohesion_publicCommunity100%
Apache-2.0

Rater Cohesion and Quality from a Vicarious Perspective

00Python, Jupyter NotebookSep 30, 20247 months agoApache-2.0

Similar Papers

Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive

82% match2023

Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive

82% match2023

Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback

81% match1 repo2024

Venire: A Machine Learning-Guided Panel Review System for Community Content Moderation

79% match2024

Towards New Benchmark for AI Alignment & Sentiment Analysis in Socially Important Issues: A Comparative Study of Human and LLMs in the Context of AGI

78% match2025

Reducing annotator bias by belief elicitation

78% match2024

Cite this paper

@article{pandita2024rater,
  title  = {Rater Cohesion and Quality from a Vicarious Perspective},
  author = {Deepak Pandita and Tharindu Cyril Weerasooriya and Sujan Dutta and Sarah K. Luger and Tharindu Ranasinghe and Ashiqur R. KhudaBukhsh and Marcos Zampieri and Christopher M. Homan},
  year   = {2024},
  eprint = {2408.08411},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.08411v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.08411.svg)](https://codeofpaper.com/paper/2408.08411)

Discussion