Evaluating German Transformer Language Models with Syntactic Agreement Tests

Karolina Zaczynska, Nils Feldhus, Robert Schwarzenberg, Aleksandra Gabryszak, Sebastian Möller
7/7/2020
cs.CL

Official code: DFKI-NLP/gevalm (7 stars, last commit 2023-06-12).

Abstract

Pre-trained transformer language models (TLMs) have recently refashioned natural language processing (NLP): Most state-of-the-art NLP models now operate on top of TLMs to benefit from contextualization and knowledge induction. To explain their success, the scientific community conducted numerous analyses. Besides other methods, syntactic agreement tests were utilized to analyse TLMs. Most of the studies were conducted for the English language, however. In this work, we analyse German TLMs. To this end, we design numerous agreement tasks, some of which consider peculiarities of the German language. Our experimental results show that state-of-the-art German TLMs generally perform well on agreement tasks, but we also identify and discuss syntactic structures that push them to their limits.

View on arXivView PDF

Code Implementations(1)

DFKI-NLP/gevalmOfficial100%

Code and data for the paper "Evaluating German Transformer Language Models with Syntactic Agreement Tests" (Zaczynska et al., 2020)

72Python, DockerfileMay 6, 20203 years ago
bertgermansyntactic-agreementtransformers

Similar Papers

Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects

85% match2026

The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

85% match2026

Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages

84% match1 repo102019

Exploring Recurrent-Depth Language Models for Multitask Clinical Text Processing

83% match2026

Multilingual Transformer Encoders: a Word-Level Task-Agnostic Evaluation

83% match2022

Knowledge-Infused Self Attention Transformers

83% match2023

Cite this paper

@article{zaczynska2020evaluating,
  title  = {Evaluating German Transformer Language Models with Syntactic Agreement Tests},
  author = {Karolina Zaczynska and Nils Feldhus and Robert Schwarzenberg and Aleksandra Gabryszak and Sebastian Möller},
  year   = {2020},
  eprint = {2007.03765},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2007.03765}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2007.03765.svg)](https://codeofpaper.com/paper/2007.03765)

Discussion