SeerAttention-R: Sparse Attention Adaptation for Long Reasoning

Yizhao Gao, Shuming Guo, Shijie Cao, Yuqing Xia, Yu Cheng +10 more
6/10/2025
cs.LGcs.AI

Official code: microsoft/SeerAttention (213 stars, MIT, last commit 2026-04-08).

Abstract

We introduce SeerAttention-R, a sparse attention framework specifically tailored for the long decoding of reasoning models. Extended from SeerAttention, SeerAttention-R retains the design of learning attention sparsity through a self-distilled gating mechanism, while removing query pooling to accommodate auto-regressive decoding. With a lightweight plug-in gating, SeerAttention-R is flexible and can be easily integrated into existing pretrained model without modifying the original parameters. We demonstrate that SeerAttention-R, trained on just 0.4B tokens, maintains near-lossless reasoning accuracy with 4K token budget in AIME benchmark under large sparse attention block sizes (64/128). Using TileLang, we develop a highly optimized sparse decoding kernel that achieves near-theoretical speedups of up to 9x over FlashAttention-3 on H100 GPU at 90% sparsity. Code is available at: https://github.com/microsoft/SeerAttention.

View on arXivView PDF

Code Implementations(1)

microsoft/SeerAttentionOfficial100%
MIT

SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs

21320C++, TeXOct 8, 20246 months agoMIT

Similar Papers

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

89% match2 repos632025

Gated Sparse Attention: Combining Computational Efficiency with Training Stability for Long-Context Language Models

87% match2026

SparseAccelerate: Efficient Long-Context Inference for Mid-Range GPUs

87% match2024

Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection

86% match1 repo102026

Less Is More: Training-Free Sparse Attention with Global Locality for Efficient Reasoning

86% match1 repo362025

Post-Training Sparse Attention with Double Sparsity

86% match1 repo1072024

Cite this paper

@article{gao2025seerattentionr,
  title  = {SeerAttention-R: Sparse Attention Adaptation for Long Reasoning},
  author = {Yizhao Gao and Shuming Guo and Shijie Cao and Yuqing Xia and Yu Cheng and Lei Wang and Lingxiao Ma and Yutao Sun and Tianzhu Ye and Li Dong and Hayden Kwok-Hay So and Yu Hua and Ting Cao and Fan Yang and Mao Yang},
  year   = {2025},
  eprint = {2506.08889},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.08889v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.08889.svg)](https://codeofpaper.com/paper/2506.08889)

Discussion