Generalized Measures of Anticipation and Responsivity in Online Language Processing

Mario Giulianelli, Andreas Opedal, Ryan Cotterell
9/16/2024
cs.CLcs.AIcs.IT

No official code found. Best implementation: rycolab/generalized-surprisal (2 stars, last commit 2024-10-16).

Abstract

We introduce a generalization of classic information-theoretic measures of predictive uncertainty in online language processing, based on the simulation of expected continuations of incremental linguistic contexts. Our framework provides a formal definition of anticipatory and responsive measures, and it equips experimenters with the tools to define new, more expressive measures beyond standard next-symbol entropy and surprisal. While extracting these standard quantities from language models is convenient, we demonstrate that using Monte Carlo simulation to estimate alternative responsive and anticipatory measures pays off empirically: New special cases of our generalized formula exhibit enhanced predictive power compared to surprisal for human cloze completion probability as well as ELAN, LAN, and N400 amplitudes, and greater complementarity with surprisal in predicting reading times.

View on arXivView PDF

Code Implementations(1)

rycolab/generalized-surprisalCommunity100%
21Python, Jupyter NotebookSep 11, 20242 years ago

Similar Papers

Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differences

82% match1 repo22024

The Effect of Surprisal on Reading Times in Information Seeking and Repeated Reading

81% match1 repo2024

Do Language Models Update their Forecasts with New Information?

81% match2025

On the scaling relationship between cloze probabilities and language model next-token prediction

80% match2026

Eliciting Uncertainty in Chain-of-Thought to Mitigate Bias against Forecasting Harmful User Behaviors

80% match2024

What do Language Model Probabilities Represent? From Distribution Estimation to Response Prediction

80% match2025

Cite this paper

@article{giulianelli2024generalized,
  title  = {Generalized Measures of Anticipation and Responsivity in Online Language Processing},
  author = {Mario Giulianelli and Andreas Opedal and Ryan Cotterell},
  year   = {2024},
  eprint = {2409.10728},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.10728v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.10728.svg)](https://codeofpaper.com/paper/2409.10728)

Discussion