Colorless green recurrent networks dream hierarchically

Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, Marco Baroni
3/29/2018
cs.CL

Official code: facebookresearch/colorlessgreenRNNs (99 stars, last commit 2021-10-12).

Abstract

Recurrent neural networks (RNNs) have achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language. We investigate here to what extent RNNs learn to track abstract hierarchical syntactic structure. We test whether RNNs trained with a generic language modeling objective in four languages (Italian, English, Hebrew, Russian) can predict long-distance number agreement in various constructions. We include in our evaluation nonsensical sentences where RNNs cannot rely on semantic or lexical cues ("The colorless green ideas I ate with the chair sleep furiously"), and, for Italian, we compare model performance to human intuitions. Our language-model-trained RNNs make reliable predictions about long-distance agreement, and do not lag much behind human performance. We thus bring support to the hypothesis that RNNs are not just shallow-pattern extractors, but they also acquire deeper grammatical competence.

View on arXivView PDF

Code Implementations(1)

facebookresearch/colorlessgreenRNNsOfficial100%

Code and data for paper Colorless Green Recurrent Networks Dream Hierarchically

9924Perl, ShellDec 11, 20175 years agoNOASSERTION

Similar Papers

Brain Score Tracks Shared Properties of Languages: Evidence from Many Natural Languages and Structured Sequences

87% match2026

Language Embeddings Sometimes Contain Typological Generalizations

85% match1 repo52023

What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure

83% match1 repo72023

Causal Graph in Language Model Rediscovers Cortical Hierarchy in Human Narrative Processing

83% match2023

From Language to Cognition: How LLMs Outgrow the Human Language Network

83% match2025

Bearing Syntactic Fruit with Stack-Augmented Neural Networks

83% match1 repo32025

Cite this paper

@article{gulordava2018colorless,
  title  = {Colorless green recurrent networks dream hierarchically},
  author = {Kristina Gulordava and Piotr Bojanowski and Edouard Grave and Tal Linzen and Marco Baroni},
  year   = {2018},
  eprint = {1803.11138},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1803.11138}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1803.11138.svg)](https://codeofpaper.com/paper/1803.11138)

Discussion