Crosslingual Structural Priming and the Pre-Training Dynamics of Bilingual Language Models

Catherine Arnett, Tyler A. Chang, James A. Michaelov, Benjamin K. Bergen
10/11/2023
cs.CL

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

Do multilingual language models share abstract grammatical representations across languages, and if so, when do these develop? Following Sinclair et al. (2022), we use structural priming to test for abstract grammatical representations with causal effects on model outputs. We extend the approach to a Dutch-English bilingual setting, and we evaluate a Dutch-English language model during pre-training. We find that crosslingual structural priming effects emerge early after exposure to the second language, with less than 1M tokens of data in that language. We discuss implications for data contamination, low-resource transfer, and how abstract grammatical representations emerge in multilingual models.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

On the Acquisition of Shared Grammatical Representations in Bilingual Language Models

93% match2025

Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models

91% match2023

Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages

87% match1 repo172025

Language Models as Artificial Learners: Investigating Crosslinguistic Influence

86% match2026

Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models

86% match1 repo12026

The Same But Different: Structural Similarities and Differences in Multilingual Language Modeling

85% match2026

Cite this paper

@article{arnett2023crosslingual,
  title  = {Crosslingual Structural Priming and the Pre-Training Dynamics of Bilingual Language Models},
  author = {Catherine Arnett and Tyler A. Chang and James A. Michaelov and Benjamin K. Bergen},
  year   = {2023},
  eprint = {2310.07929},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.07929v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.07929.svg)](https://codeofpaper.com/paper/2310.07929)

Discussion