QuALITY: Question Answering with Long Input Texts, Yes!

Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang +6 more
12/16/2021
cs.CL

Official code: nyu-mll/quality (155 stars, last commit 2025-01-17).

Abstract

To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our questions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addition, only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Our baseline models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%).

View on arXivView PDF

Code Implementations(1)

nyu-mll/qualityOfficial100%
15512PythonDec 13, 20211 years ago

Similar Papers

NarrativeXL: A Large-scale Dataset For Long-Term Memory Models

88% match1 repo102023

LooGLE: Can Long-Context Language Models Understand Long Contexts?

86% match1 repo2002023

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

85% match2026

Marathon: A Race Through the Realm of Long Context with Large Language Models

85% match1 repo102023

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

85% match1 repo1.2k2023

BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models

84% match1 repo362023

Cite this paper

@article{pang2021quality,
  title  = {QuALITY: Question Answering with Long Input Texts, Yes!},
  author = {Richard Yuanzhe Pang and Alicia Parrish and Nitish Joshi and Nikita Nangia and Jason Phang and Angelica Chen and Vishakh Padmakumar and Johnny Ma and Jana Thompson and He He and Samuel R. Bowman},
  year   = {2021},
  eprint = {2112.08608},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2112.08608}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2112.08608.svg)](https://codeofpaper.com/paper/2112.08608)

Discussion