Replacing softmax with ReLU in Vision Transformers

Mitchell Wortsman, Jaehoon Lee, Justin Gilmer, Simon Kornblith
9/15/2023
cs.CVcs.LG

No confident code implementation found for this paper.

Abstract

Previous research observed accuracy degradation when replacing the attention softmax with a point-wise activation such as ReLU. In the context of vision transformers, we find that this degradation is mitigated when dividing by sequence length. Our experiments training small to large vision transformers on ImageNet-21k indicate that ReLU-attention can approach or match the performance of softmax-attention in terms of scaling behavior as a function of compute.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Theory, Analysis, and Best Practices for Sigmoid Self-Attention

85% match1 repo3112024

Rethinking Attention: Polynomial Alternatives to Softmax in Transformers

84% match2024

SeTformer is What You Need for Vision and Language

83% match2024

A Study on ReLU and Softmax in Transformer

83% match2023

Why Softmax Attention Outperforms Linear Attention

83% match2023

Three things everyone should know about Vision Transformers

83% match1 repo4.3k2022

Cite this paper

@article{wortsman2023replacing,
  title  = {Replacing softmax with ReLU in Vision Transformers},
  author = {Mitchell Wortsman and Jaehoon Lee and Justin Gilmer and Simon Kornblith},
  year   = {2023},
  eprint = {2309.08586},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.08586v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.08586.svg)](https://codeofpaper.com/paper/2309.08586)

Discussion