Assessment of Pre-Trained Models Across Languages and Grammars

Alberto Muñoz-Ortiz, David Vilares, Carlos Gómez-Rodríguez
9/20/2023
cs.CL

No official code found. Best implementation: amunozo/multilingual-assessment (0 stars, MIT, last commit 2026-01-20).

Abstract

We present an approach for assessing how multilingual large language models (LLMs) learn syntax in terms of multi-formalism syntactic structures. We aim to recover constituent and dependency structures by casting parsing as sequence labeling. To do so, we select a few LLMs and study them on 13 diverse UD treebanks for dependency parsing and 10 treebanks for constituent parsing. Our results show that: (i) the framework is consistent across encodings, (ii) pre-trained word vectors do not favor constituency representations of syntax over dependencies, (iii) sub-word tokenization is needed to represent syntax, in contrast to character-based models, and (iv) occurrence of a language in the pretraining data is more important than the amount of task data when recovering syntax from the word vectors.

View on arXivView PDF

Code Implementations(1)

amunozo/multilingual-assessmentCommunity100%
MIT
00Python, Jupyter NotebookSep 6, 20239 months agoMIT

Similar Papers

Parsing as Pretraining

86% match1 repo162020

Arabic Morphosyntactic Tagging and Dependency Parsing with Large Language Models

85% match2026

Probing LLMs for Joint Encoding of Linguistic Categories

85% match1 repo72023

To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs

84% match2026

Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models

83% match2023

JCoLA: Japanese Corpus of Linguistic Acceptability

83% match1 repo192023

Cite this paper

@article{muozortiz2023assessment,
  title  = {Assessment of Pre-Trained Models Across Languages and Grammars},
  author = {Alberto Muñoz-Ortiz and David Vilares and Carlos Gómez-Rodríguez},
  year   = {2023},
  eprint = {2309.11165},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.11165v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.11165.svg)](https://codeofpaper.com/paper/2309.11165)

Discussion