Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models

Yang Liu, Hongming Li, Melissa Xiaohui Qin, Qiankun Liu, Chao Huang
4/17/2026
cs.CL

Official code: jacklanda/SemanticQA (10 stars, MIT, last commit 2026-04-26).

Abstract

We present SemanticQA, an evaluation suite designed to assess language models (LMs) in semantic phrase processing tasks. The benchmark consolidates existing multiword expression (MwE) resources and reorganizes them into a unified testbed. It covers both general lexical phenomena, such as lexical collocations, and three fine-grained categories: idiomatic expressions, noun compounds, and verbal constructions. Through SemanticQA, we assess LMs of diverse architectures and scales in extraction, classification, and interpretation tasks, as well as sequential task compositions. We reveal substantial performance variation, particularly on tasks requiring semantic reasoning, highlighting differences in reasoning efficacy and semantic understanding of LMs, providing insights for pushing LMs with stronger comprehension on non-trivial semantic phrases. The evaluation harness and data of SemanticQA are available at https://github.com/jacklanda/SemanticQA.

View on arXivView PDF

Code Implementations(1)

jacklanda/SemanticQAOfficial100%
MIT

[ACL 2026] Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models

101MDX, HTMLOct 28, 20235 months agoMIT
benchmarkreasoningsemantics

Similar Papers

Revisiting a Pain in the Neck: Semantic Phrase Processing Benchmark for Language Models

90% match2024

Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models

83% match1 repo612024

3D-EX : A Unified Dataset of Definitions and Dictionary Examples

82% match1 repo52023

To What Extent Do Large Language Models Understand Bangla Idioms?

82% match2026

The Two Word Test: A Semantic Benchmark for Large Language Models

82% match1 repo72023

XSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations

82% match1 repo92023

Cite this paper

@article{liu2026revisiting,
  title  = {Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models},
  author = {Yang Liu and Hongming Li and Melissa Xiaohui Qin and Qiankun Liu and Chao Huang},
  year   = {2026},
  eprint = {2604.16593},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2604.16593v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2604.16593.svg)](https://codeofpaper.com/paper/2604.16593)

Discussion