ConvNets Match Vision Transformers at Scale

Samuel L. Smith, Andrew Brock, Leonard Berrada, Soham De
10/25/2023
cs.CVcs.LGcs.NE

No official code found. Best implementation: kyegomez/ConvNet (6 stars, MIT, last commit 2024-09-09).

Abstract

Many researchers believe that ConvNets perform well on small or moderately sized datasets, but are not competitive with Vision Transformers when given access to datasets on the web-scale. We challenge this belief by evaluating a performant ConvNet architecture pre-trained on JFT-4B, a large labelled dataset of images often used for training foundation models. We consider pre-training compute budgets between 0.4k and 110k TPU-v4 core compute hours, and train a series of networks of increasing depth and width from the NFNet model family. We observe a log-log scaling law between held out loss and compute budget. After fine-tuning on ImageNet, NFNets match the reported performance of Vision Transformers with comparable compute budgets. Our strongest fine-tuned model achieves a Top-1 accuracy of 90.4%.

View on arXivView PDF

Code Implementations(1)

kyegomez/ConvNetCommunity65%
MIT

Implementation of the NFNets from the paper: "ConvNets Match Vision Transformers at Scale" by Google Research

60PythonOct 26, 20232 years agoMIT
aiconvolutional-layersconvolutional-neural-networksdeeplearningmachine-learning+3 more

Similar Papers

Convolutional Neural Nets vs Vision Transformers: A SpaceNet Case Study with Balanced vs Imbalanced Regimes

85% match2025

ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy

85% match1 repo1032026

Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?

83% match1 repo382023

Convolutional Initialization for Data-Efficient Vision Transformers

82% match1 repo152024

ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameters

82% match1 repo32025

Structured Initialization for Attention in Vision Transformers

82% match1 repo42024

Cite this paper

@article{smith2023convnets,
  title  = {ConvNets Match Vision Transformers at Scale},
  author = {Samuel L. Smith and Andrew Brock and Leonard Berrada and Soham De},
  year   = {2023},
  eprint = {2310.16764},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.16764v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.16764.svg)](https://codeofpaper.com/paper/2310.16764)

Discussion