Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation

Nan-Jiang Jiang, Chenhao Tan, Marie-Catherine de Marneffe
4/24/2023
cs.CL

No confident code implementation found for this paper.

Abstract

Human label variation (Plank 2022), or annotation disagreement, exists in many natural language processing (NLP) tasks. To be robust and trusted, NLP models need to identify such variation and be able to explain it. To this end, we created the first ecologically valid explanation dataset with diverse reasoning, LiveNLI. LiveNLI contains annotators' highlights and free-text explanations for the label(s) of their choice for 122 English Natural Language Inference items, each with at least 10 annotations. We used its explanations for chain-of-thought prompting, and found there is still room for improvement in GPT-3's ability to predict label distribution with in-context learning.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Ecologically Valid Explanations for Label Variation in NLI

97% match2023

Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations

88% match2025

SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMs

86% match2026

LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference

86% match1 repo12025

From Ground Truth to Measurement: A Statistical Framework for Human Labeling

85% match2026

Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information

85% match1 repo62023

Cite this paper

@article{jiang2023understanding,
  title  = {Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation},
  author = {Nan-Jiang Jiang and Chenhao Tan and Marie-Catherine de Marneffe},
  year   = {2023},
  eprint = {2304.12443},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.12443v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.12443.svg)](https://codeofpaper.com/paper/2304.12443)

Discussion