Flash Window Attention: speedup the attention computation for Swin Transformer

Zhendong Zhang
1/11/2025
cs.CV

No official code found. Best implementation: zzd1992/FlashWindowAttention (31 stars, last commit 2025-06-14).

Abstract

To address the high resolution of image pixels, the Swin Transformer introduces window attention. This mechanism divides an image into non-overlapping windows and restricts attention computation to within each window, significantly enhancing computational efficiency. To further optimize this process, one might consider replacing standard attention with flash attention, which has proven to be more efficient in language models. However, a direct substitution is ineffective. Flash attention is designed for long sequences, whereas window attention deals with shorter sequences but must handle numerous of them in parallel. In this report, we present an optimized solution called Flash Window Attention, tailored specifically for window attention. Flash Window Attention improves attention computation efficiency by up to 300% and enhances end-to-end runtime efficiency by up to 30%. Our code is available online.

View on arXivView PDF

Code Implementations(1)

zzd1992/FlashWindowAttentionCommunity100%

Speedup the attention computation of Swin Transformer

314PythonJan 11, 20251 years ago

Similar Papers

Emulating Self-attention with Convolution for Efficient Image Super-Resolution

86% match1 repo1132025

A-VL: Adaptive Attention for Large Vision-Language Models

86% match1 repo22024

The Linear Attention Resurrection in Vision Transformer

86% match1 repo162025

FwNet-ECA: A Classification Model Enhancing Window Attention with Global Receptive Fields via Fourier Filtering Operations

85% match1 repo22025

BlindSight: Harnessing Sparsity for Efficient Vision-Language Models

85% match2025

DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformers

84% match2025

Cite this paper

@article{zhang2025flash,
  title  = {Flash Window Attention: speedup the attention computation for Swin Transformer},
  author = {Zhendong Zhang},
  year   = {2025},
  eprint = {2501.06480},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2501.06480v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2501.06480.svg)](https://codeofpaper.com/paper/2501.06480)

Discussion