LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs

Jianghao Chen, Junhong Wu, Yangyifan Xu, Jiajun Zhang
3/4/2025
cs.CL

No official code found. Best implementation: ZNLP/LADM (9 stars, last commit 2025-06-12).

Abstract

Long-context modeling has drawn more and more attention in the area of Large Language Models (LLMs). Continual training with long-context data becomes the de-facto method to equip LLMs with the ability to process long inputs. However, it still remains an open challenge to measure the quality of long-context training data. To address this issue, we propose a Long-context data selection framework with Attention-based Dependency Measurement (LADM), which can efficiently identify high-quality long-context data from a large-scale, multi-domain pre-training corpus. LADM leverages the retrieval capabilities of the attention mechanism to capture contextual dependencies, ensuring a comprehensive quality measurement of long-context data. Experimental results show that our LADM framework significantly boosts the performance of LLMs on multiple long-context tasks with only 1B tokens for continual training.

View on arXivView PDF

Code Implementations(1)

ZNLP/LADMCommunity68%

[ACL 2025] LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs

90Shell, PythonJun 12, 20251 years ago

Similar Papers

Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data

91% match2025

Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models

90% match1 repo612024

LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models

89% match2024

A Controlled Study on Long Context Extension and Generalization in LLMs

88% match1 repo652024

SEAL: Scaling to Emphasize Attention for Long-Context Retrieval

88% match2025

How to Train Long-Context Language Models (Effectively)

87% match1 repo2622024

Cite this paper

@article{chen2025ladm,
  title  = {LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs},
  author = {Jianghao Chen and Junhong Wu and Yangyifan Xu and Jiajun Zhang},
  year   = {2025},
  eprint = {2503.02502},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2503.02502v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2503.02502.svg)](https://codeofpaper.com/paper/2503.02502)

Discussion