AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention

Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun +5 more
4/9/2026
cs.CL

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

Long-context inference in LLMs faces the dual challenges of quadratic attention complexity and prohibitive KV cache memory. While token-level sparse attention offers superior accuracy, its indexing overhead is costly; block-level methods improve efficiency but sacrifice precision. We propose AsyncTLS, a hierarchical sparse attention system that combines coarse-grained block filtering with fine-grained token selection to balance accuracy and efficiency, coupled with an asynchronous offloading engine that overlaps KV cache transfers with computation via temporal locality exploitation. Evaluated on Qwen3 and GLM-4.7-Flash across GQA, and MLA architectures, AsyncTLS achieves accuracy comparable to full attention while delivering 1.2x - 10.0x operator speedups and 1.3x - 4.7x end-to-end throughput improvements on 48k - 96k contexts.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

SPIN: Shadow Predictive Indexer for Sparse Attention

91% match2026

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

89% match1 repo92026

You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

89% match2026

CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference

89% match2026

Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decoding

88% match1 repo2026

HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention

88% match2026

Cite this paper

@article{hu2026asynctls,
  title  = {AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention},
  author = {Yuxuan Hu and Jianchao Tan and Jiaqi Zhang and Wen Zan and Pingwei Sun and Yifan Lu and Yerui Sun and Yuchen Xie and Xunliang Cai and Jing Zhang},
  year   = {2026},
  eprint = {2604.07815},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2604.07815v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2604.07815.svg)](https://codeofpaper.com/paper/2604.07815)

Discussion