Linguini: A benchmark for language-agnostic linguistic reasoning

Eduardo Sánchez, Belen Alastruey, Christophe Ropers, Pontus Stenetorp, Mikel Artetxe +1 more
9/18/2024
cs.CL

No official code found. Best implementation: facebookresearch/linguini (10 stars, CC-BY-SA-4.0, last commit 2024-09-19).

Abstract

We propose a new benchmark to measure a language model's linguistic reasoning skills without relying on pre-existing language-specific knowledge. The test covers 894 questions grouped in 160 problems across 75 (mostly) extremely low-resource languages, extracted from the International Linguistic Olympiad corpus. To attain high accuracy on this benchmark, models don't need previous knowledge of the tested language, as all the information needed to solve the linguistic puzzle is presented in the context. We find that, while all analyzed models rank below 25% accuracy, there is a significant gap between open and closed models, with the best-performing proprietary model at 24.05% and the best-performing open model at 8.84%.

View on arXivView PDF

Code Implementations(1)

facebookresearch/linguiniCommunity100%
CC-BY-SA-4.0

Linguini is a benchmark to measure a language model’s linguistic reasoning skills without relying on pre-existing language-specific knowledge, based on the International Linguistic Olympiad problems.

101Sep 18, 20242 years agoCC-BY-SA-4.0

Similar Papers

LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs

88% match2025

LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages

88% match1 repo122026

modeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models

87% match2024

LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation

87% match2025

IOLBENCH: Benchmarking LLMs on Linguistic Reasoning

86% match1 repo42025

AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models

84% match1 repo102025

Cite this paper

@article{snchez2024linguini,
  title  = {Linguini: A benchmark for language-agnostic linguistic reasoning},
  author = {Eduardo Sánchez and Belen Alastruey and Christophe Ropers and Pontus Stenetorp and Mikel Artetxe and Marta R. Costa-jussà},
  year   = {2024},
  eprint = {2409.12126},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.12126v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.12126.svg)](https://codeofpaper.com/paper/2409.12126)

Discussion