Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State

Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros +1 more
3/8/2019
cs.CL

Official code: langprocgroup/nn_syntactic_state (17 stars, last commit 2019-03-04).

Abstract

We deploy the methods of controlled psycholinguistic experimentation to shed light on the extent to which the behavior of neural network language models reflects incremental representations of syntactic state. To do so, we examine model behavior on artificial sentences containing a variety of syntactically complex structures. We test four models: two publicly available LSTM sequence models of English (Jozefowicz et al., 2016; Gulordava et al., 2018) trained on large datasets; an RNNG (Dyer et al., 2016) trained on a small, parsed dataset; and an LSTM trained on the same small corpus as the RNNG. We find evidence that the LSTMs trained on large datasets represent syntactic state over large spans of text in a way that is comparable to the RNNG, while the LSTM trained on the small dataset does not or does so only weakly.

View on arXivView PDF

Code Implementations(1)

langprocgroup/nn_syntactic_stateOfficial100%

Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State

171R, ShellMar 4, 20197 years ago

Similar Papers

JCoLA: Japanese Corpus of Linguistic Acceptability

85% match1 repo192023

Evaluating Neural Language Models as Cognitive Models of Language Acquisition

85% match1 repo2023

Brain Score Tracks Shared Properties of Languages: Evidence from Many Natural Languages and Structured Sequences

84% match2026

Can training neural language models on a curriculum with developmentally plausible data improve alignment with human reading behavior?

84% match2023

Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution

84% match1 repo72023

Mapping Brains with Language Models: A Survey

83% match2023

Cite this paper

@article{futrell2019neural,
  title  = {Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State},
  author = {Richard Futrell and Ethan Wilcox and Takashi Morita and Peng Qian and Miguel Ballesteros and Roger Levy},
  year   = {2019},
  eprint = {1903.03260},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1903.03260}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1903.03260.svg)](https://codeofpaper.com/paper/1903.03260)

Discussion