Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics

James A. Michaelov, Catherine Arnett, Benjamin K. Bergen
4/30/2024
cs.CL

No official code found. Best implementation: jmichaelov/recurrent-vs-transformer-modeling (3 stars, MIT, last commit 2024-08-03).

Abstract

Transformers have generally supplanted recurrent neural networks as the dominant architecture for both natural language processing tasks and for modelling the effect of predictability on online human language comprehension. However, two recently developed recurrent model architectures, RWKV and Mamba, appear to perform natural language tasks comparably to or better than transformers of equivalent scale. In this paper, we show that contemporary recurrent models are now also able to match - and in some cases, exceed - the performance of comparably sized transformers at modeling online human language comprehension. This suggests that transformer language models are not uniquely suited to this task, and opens up new directions for debates about the extent to which architectural features of language models make them better or worse models of human language comprehension.

View on arXivView PDF

Code Implementations(1)

jmichaelov/recurrent-vs-transformer-modelingCommunity100%
MIT

Repository for 'Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics', to be presented at COLM 2024.

30R, ShellAug 3, 20242 years agoMIT

Similar Papers

Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language models

86% match2025

Does Transformer Interpretability Transfer to RNNs?

86% match2 repos332024

Modeling Bilingual Sentence Processing: Evaluating RNN and Transformer Architectures for Cross-Language Structural Priming

86% match2024

Why transformers are obviously good models of language

86% match2024

Autoregressive + Chain of Thought = Recurrent: Recurrence's Role in Language Models' Computability and a Revisit of Recurrent Transformer

85% match2024

Beyond Human-Like Processing: Large Language Models Perform Equivalently on Forward and Backward Scientific Text

85% match2024

Cite this paper

@article{michaelov2024revenge,
  title  = {Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics},
  author = {James A. Michaelov and Catherine Arnett and Benjamin K. Bergen},
  year   = {2024},
  eprint = {2404.19178},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.19178v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.19178.svg)](https://codeofpaper.com/paper/2404.19178)

Discussion