LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition

Youbing Hu, Yun Cheng, Anqi Lu, Zhiqiang Cao, Dawei Wei +2 more
1/8/2024
cs.CVcs.AI

Official code: edgeai1/LF-ViT (11 stars, last commit 2024-09-23).

Abstract

The Vision Transformer (ViT) excels in accuracy when handling high-resolution images, yet it confronts the challenge of significant spatial redundancy, leading to increased computational and memory requirements. To address this, we present the Localization and Focus Vision Transformer (LF-ViT). This model operates by strategically curtailing computational demands without impinging on performance. In the Localization phase, a reduced-resolution image is processed; if a definitive prediction remains elusive, our pioneering Neighborhood Global Class Attention (NGCA) mechanism is triggered, effectively identifying and spotlighting class-discriminative regions based on initial findings. Subsequently, in the Focus phase, this designated region is used from the original image to enhance recognition. Uniquely, LF-ViT employs consistent parameters across both phases, ensuring seamless end-to-end optimization. Our empirical tests affirm LF-ViT's prowess: it remarkably decreases Deit-S's FLOPs by 63\% and concurrently amplifies throughput twofold. Code of this project is at https://github.com/edgeai1/LF-ViT.git.

View on arXivView PDF

Code Implementations(1)

edgeai1/LF-ViTOfficial100%

This is Pytorch implementation of our paper "LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition".

111PythonDec 11, 20232 years ago

Similar Papers

LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate

86% match1 repo62026

HAAT: Hybrid Attention Aggregation Transformer for Image Super-Resolution

85% match1 repo5302024

Looking Locally: Object-Centric Vision Transformers as Foundation Models for Efficient Segmentation

85% match1 repo32025

LMLT: Low-to-high Multi-Level Vision Transformer for Image Super-Resolution

85% match1 repo322024

Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures

85% match2 repos5492026

HiTSR: A Hierarchical Transformer for Reference-based Super-Resolution

85% match1 repo62024

Cite this paper

@article{hu2024lfvit,
  title  = {LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition},
  author = {Youbing Hu and Yun Cheng and Anqi Lu and Zhiqiang Cao and Dawei Wei and Jie Liu and Zhijun Li},
  year   = {2024},
  eprint = {2402.00033},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.00033v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.00033.svg)](https://codeofpaper.com/paper/2402.00033)

Discussion