DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration

Hanzhi Zhang, Heng Fan, Kewei Sha, Yan Huang, Yunhe Feng
6/6/2025
cs.CLcs.AI

No official code found. Best implementation: ResponsibleAILab/DAM (14 stars, last commit 2025-06-16).

Abstract

Long-context understanding is crucial for many NLP applications, yet transformers struggle with efficiency due to the quadratic complexity of self-attention. Sparse attention methods alleviate this cost but often impose static, predefined masks, failing to capture heterogeneous attention patterns. This results in suboptimal token interactions, limiting adaptability and retrieval accuracy in long-sequence tasks. This work introduces a dynamic sparse attention mechanism that assigns adaptive masks at the attention-map level, preserving heterogeneous patterns across layers and heads. Unlike existing approaches, our method eliminates the need for fine-tuning and predefined mask structures while maintaining computational efficiency. By learning context-aware attention structures, it achieves high alignment with full-attention models, ensuring minimal performance degradation while reducing memory and compute overhead. This approach provides a scalable alternative to full attention, enabling the practical deployment of large-scale Large Language Models (LLMs) without sacrificing retrieval performance. DAM is available at: https://github.com/HanzhiZhang-Ulrica/DAM.

View on arXivView PDF

Code Implementations(1)

ResponsibleAILab/DAMCommunity100%

Dynamic Attention Mask (DAM) generate adaptive sparse attention masks per layer and head for Transformer models, enabling long-context inference with lower compute and memory overhead without fine-tuning.

141PythonMay 27, 20251 years ago
efficient-aiinference-optimizationsparse-attention

Similar Papers

Trainable Dynamic Mask Sparse Attention

92% match2025

Training-free Context-adaptive Attention for Efficient Long Context Modeling

90% match2025

RRAttention: Dynamic Block Sparse Attention via Per-Head Round-Robin Shifts for Long-Context Inference

89% match2026

Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection

88% match1 repo102026

PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention

88% match1 repo22025

Ltri-LLM: Streaming Long Context Inference for LLMs with Training-Free Dynamic Triangular Attention Pattern

88% match2024

Cite this paper

@article{zhang2025dam,
  title  = {DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration},
  author = {Hanzhi Zhang and Heng Fan and Kewei Sha and Yan Huang and Yunhe Feng},
  year   = {2025},
  eprint = {2506.11104},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.11104v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.11104.svg)](https://codeofpaper.com/paper/2506.11104)

Discussion