MultiWikiQA: A Reading Comprehension Benchmark in 300+ Languages

Dan Saattrup Smart
9/4/2025
cs.CL

No confident code implementation found for this paper.

Abstract

We introduce a new reading comprehension dataset, dubbed MultiWikiQA, which covers 306 languages. The context data comes from Wikipedia articles, with questions generated by an LLM and the answers appearing verbatim in the Wikipedia articles. We conduct a crowdsourced human evaluation of the fluency of the generated questions across 30 of the languages, providing evidence that the questions are of good quality. We evaluate 6 different language models, both decoder and encoder models of varying sizes, showing that the benchmark is sufficiently difficult and that there is a large performance discrepancy amongst the languages. The dataset and survey evaluations are freely available.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment

85% match1 repo142023

ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models

85% match2025

M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models

85% match1 repo1052026

The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks

84% match2025

Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQ

84% match1 repo52024

BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models

84% match1 repo292025

Cite this paper

@article{smart2025multiwikiqa,
  title  = {MultiWikiQA: A Reading Comprehension Benchmark in 300+ Languages},
  author = {Dan Saattrup Smart},
  year   = {2025},
  eprint = {2509.04111},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2509.04111v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2509.04111.svg)](https://codeofpaper.com/paper/2509.04111)

Discussion