How to Compute the Probability of a Word

Tiago Pimentel, Clara Meister
6/20/2024
cs.CL

No official code found. Best implementation: tpimentelms/probability-of-a-word (12 stars, MIT, last commit 2024-10-14). 1 more below.

Abstract

Language models (LMs) estimate a probability distribution over strings in a natural language; these distributions are crucial for computing perplexity and surprisal in linguistics research. While we are usually concerned with measuring these values for words, most LMs operate over subwords. Despite seemingly straightforward, accurately computing probabilities over one unit given probabilities over the other requires care. Indeed, we show here that many recent linguistic studies have been incorrectly computing these values. This paper derives the correct methods for computing word probabilities, highlighting issues when relying on language models that use beginning-of-word (bow)-marking tokenisers, e.g., the GPT family. Empirically, we show that correcting the widespread bug in probability computations affects measured outcomes in sentence comprehension and lexical optimisation analyses.

View on arXivView PDF

Code Implementations(2)

tpimentelms/probability-of-a-wordCommunity100%
MIT

Code to compute a word's probability using the fixes from "How to Compute the Probability of a Word"

126PythonJul 2, 20242 years agoMIT
tpimentelms/probability-of-a-word-experimentsCommunity65%
MIT

Code to run experiments of paper "How to Compute the Probability of a Word"

31R, ShellOct 4, 20241 years agoMIT

Similar Papers

What Can String Probability Tell Us About Grammaticality?

88% match2025

What do Language Model Probabilities Represent? From Distribution Estimation to Response Prediction

86% match2025

What Are the Odds? Language Models Are Capable of Probabilistic Reasoning

86% match1 repo62024

Large Language Model probabilities cannot distinguish between possible and impossible language

84% match2025

GPT as a Monte Carlo Language Tree: A Probabilistic Perspective

84% match1 repo462025

CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Models

84% match2024

Cite this paper

@article{pimentel2024compute,
  title  = {How to Compute the Probability of a Word},
  author = {Tiago Pimentel and Clara Meister},
  year   = {2024},
  eprint = {2406.14561},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.14561v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.14561.svg)](https://codeofpaper.com/paper/2406.14561)

Discussion