Linear Recency Bias During Training Improves Transformers' Fit to Reading Times

Christian Clark, Byung-Doh Oh, William Schuler
9/17/2024
cs.CL

No official code found. Best implementation: christian-clark/recency-bias (1 star, last commit 2025-07-01).

Abstract

Recent psycholinguistic research has compared human reading times to surprisal estimates from language models to study the factors shaping human sentence processing difficulty. Previous studies have shown a strong fit between surprisal values from Transformers and reading times. However, standard Transformers work with a lossless representation of the entire previous linguistic context, unlike models of human language processing that include memory decay. To bridge this gap, this paper evaluates a modification of the Transformer model that uses ALiBi (Press et al., 2022), a recency bias added to attention scores. Surprisal estimates with ALiBi show an improved fit to human reading times compared to a standard Transformer baseline. A subsequent analysis of attention heads suggests that ALiBi's mixture of slopes -- which determine the rate of memory decay in each attention head -- may play a role in the improvement by helping models with ALiBi to track different kinds of linguistic dependencies.

View on arXivView PDF

Code Implementations(1)

christian-clark/recency-biasCommunity56%

Code and instructions for replicating experiments from "Linear Recency Bias During Training Improves Transformers’ Fit to Reading Times" (Clark et al., COLING 2025).

10Shell, PythonDec 14, 20241 years ago

Similar Papers

Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language models

85% match2025

Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times

85% match1 repo232024

Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics

84% match1 repo32024

Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorly

83% match2025

N-gram-like Language Models Predict Reading Time Best

83% match2026

The Effect of Surprisal on Reading Times in Information Seeking and Repeated Reading

83% match1 repo2024

Cite this paper

@article{clark2024linear,
  title  = {Linear Recency Bias During Training Improves Transformers' Fit to Reading Times},
  author = {Christian Clark and Byung-Doh Oh and William Schuler},
  year   = {2024},
  eprint = {2409.11250},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.11250v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.11250.svg)](https://codeofpaper.com/paper/2409.11250)

Discussion