Understanding Why ViT Trains Badly on Small Datasets: An Intuitive Perspective

Haoran Zhu, Boyuan Chen, Carter Yang
2/7/2023
cs.CVcs.LG

No official code found. Best implementation: kentaroy47/vision-transformers-cifar10 (719 stars, MIT, last commit 2025-11-20).

Abstract

Vision transformer (ViT) is an attention neural network architecture that is shown to be effective for computer vision tasks. However, compared to ResNet-18 with a similar number of parameters, ViT has a significantly lower evaluation accuracy when trained on small datasets. To facilitate studies in related fields, we provide a visual intuition to help understand why it is the case. We first compare the performance of the two models and confirm that ViT has less accuracy than ResNet-18 when trained on small datasets. We then interpret the results by showing attention map visualization for ViT and feature map visualization for ResNet-18. The difference is further analyzed through a representation similarity perspective. We conclude that the representation of ViT trained on small datasets is hugely different from ViT trained on large datasets, which may be the reason why the performance drops a lot on small datasets.

View on arXivView PDF

Code Implementations(1)

kentaroy47/vision-transformers-cifar10Community100%
MIT

Let's train vision transformers (ViT) for cifar 10 / cifar 100!

719138PythonOct 27, 202011 months agoMIT

Similar Papers

Evaluating Graphical Perception Capabilities of Vision Transformers

84% match2026

On the Surprising Effectiveness of Attention Transfer for Vision Transformers

84% match1 repo242024

How Does Attention Work in Vision Transformers? A Visual Analytics Attempt

83% match2023

A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity

83% match2023

Powerful Design of Small Vision Transformer on CIFAR10

83% match1 repo32025

Structured Initialization for Attention in Vision Transformers

83% match1 repo42024

Cite this paper

@article{zhu2023understanding,
  title  = {Understanding Why ViT Trains Badly on Small Datasets: An Intuitive Perspective},
  author = {Haoran Zhu and Boyuan Chen and Carter Yang},
  year   = {2023},
  eprint = {2302.03751},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.03751v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.03751.svg)](https://codeofpaper.com/paper/2302.03751)

Discussion