Structured Initialization for Attention in Vision Transformers

Jianqiao Zheng, Xueqian Li, Simon Lucey
4/1/2024
cs.CV

No official code found. Best implementation: osiriszjq/structured_init (4 stars, MIT, last commit 2025-12-09).

Abstract

The training of vision transformer (ViT) networks on small-scale datasets poses a significant challenge. By contrast, convolutional neural networks (CNNs) have an architectural inductive bias enabling them to perform well on such problems. In this paper, we argue that the architectural bias inherent to CNNs can be reinterpreted as an initialization bias within ViT. This insight is significant as it empowers ViTs to perform equally well on small-scale problems while maintaining their flexibility for large-scale applications. Our inspiration for this ``structured'' initialization stems from our empirical observation that random impulse filters can achieve comparable performance to learned filters within CNNs. Our approach achieves state-of-the-art performance for data-efficient ViT learning across numerous benchmarks including CIFAR-10, CIFAR-100, and SVHN.

View on arXivView PDF

Code Implementations(1)

osiriszjq/structured_initCommunity100%
MIT

Structured Initialization for Attention in Vision Transformers

40MDX, ShellMar 29, 202410 months agoMIT
attention-mechanismconvmixerinitializationtransformers

Similar Papers

Convolutional Initialization for Data-Efficient Vision Transformers

96% match1 repo152024

The Overfocusing Bias of Convolutional Neural Networks: A Saliency-Guided Regularization Approach

88% match2024

IBiT: Utilizing Inductive Biases to Create a More Data Efficient Attention Mechanism

87% match2025

The Power of Linear Combinations: Learning with Random Convolutions

86% match2023

Cumulative Spatial Knowledge Distillation for Vision Transformers

86% match1 repo152026

Simple Self Organizing Map with Vision Transformers

85% match2025

Cite this paper

@article{zheng2024structured,
  title  = {Structured Initialization for Attention in Vision Transformers},
  author = {Jianqiao Zheng and Xueqian Li and Simon Lucey},
  year   = {2024},
  eprint = {2404.01139},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.01139v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.01139.svg)](https://codeofpaper.com/paper/2404.01139)

Discussion