EXECUTE: A Multilingual Benchmark for LLM Token Understanding

Lukas Edman, Helmut Schmid, Alexander Fraser
5/23/2025
cs.CL

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE. Our simplified framework allows easy expansion to any language. Tests across multiple LLMs reveal that challenges in other languages are not always on the character level as in English. Some languages show word-level processing issues, some show no issues at all. We also examine sub-character tasks in Chinese, Japanese, and Korean to assess LLMs' understanding of character components.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Cite this paper

@article{edman2025execute,
  title  = {EXECUTE: A Multilingual Benchmark for LLM Token Understanding},
  author = {Lukas Edman and Helmut Schmid and Alexander Fraser},
  year   = {2025},
  eprint = {2505.17784},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.17784v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.17784.svg)](https://codeofpaper.com/paper/2505.17784)

Discussion