LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework

Zecheng Tang, Haitian Wang, Quantong Qiu, Baibei Ji, Ruoxi Sun +3 more
7/7/2025
cs.CL

No official code found. Best implementation: LCM-Lab/LOOM-Eval (31 stars, MIT, last commit 2026-02-25).

Abstract

Long-context processing has become a fundamental capability for large language models~(LLMs). To assess model's long-context performance, numerous long-context evaluation benchmarks have been proposed. However, variations in evaluation settings across these benchmarks lead to inconsistent results, making it difficult to draw reliable comparisons. Besides, the high computational cost of long-context evaluation poses a significant barrier for the community to conduct comprehensive assessments of long-context models. In this paper, we propose LOOM-Scope, a comprehensive and efficient framework for long-context evaluation. LOOM-Scope standardizes evaluation settings across diverse benchmarks, supports deployment of efficient long-context inference acceleration methods, and introduces a holistic yet lightweight benchmark suite to evaluate models comprehensively. Homepage: https://loomscope.github.io

View on arXivView PDF

Code Implementations(1)

LCM-Lab/LOOM-EvalCommunity100%
MIT

A comprehensive and efficient long-context model evaluation framework

314C, C++Jun 30, 20257 months agoMIT
benchmark-suitelong-context-evaluation

Similar Papers

HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly

89% match2 repos2232024

A Comprehensive Survey on Long Context Language Modeling

89% match1 repo2492025

100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

88% match1 repo72025

A Controllable Examination for Long-Context Language Models

88% match2025

A Controlled Study on Long Context Extension and Generalization in LLMs

87% match1 repo652024

LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion

87% match2025

Cite this paper

@article{tang2025loomscope,
  title  = {LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework},
  author = {Zecheng Tang and Haitian Wang and Quantong Qiu and Baibei Ji and Ruoxi Sun and Keyan Zhou and Juntao Li and Min Zhang},
  year   = {2025},
  eprint = {2507.04723},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2507.04723v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2507.04723.svg)](https://codeofpaper.com/paper/2507.04723)

Discussion