Transformer Approximations from ReLUs

Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee, Han Liu
4/27/2026
cs.LGcs.AIstat.ML

No confident code implementation found for this paper.

Abstract

We provide a systematic recipe for translating ReLU approximation results to softmax attention mechanism. This recipe covers many common approximation targets. Importantly, it yields target-specific, economic resource bounds beyond universal approximation statements. We showcase the recipe on multiplication, reciprocal computation, and min/max primitives. These results provide new analytical tools for analyzing softmax transformer models.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Rethinking Attention: Polynomial Alternatives to Softmax in Transformers

86% match2024

Self-Adjust Softmax

85% match2025

Vertex-Softmax: Tight Transformer Verification via Exact Softmax Optimization

84% match2026

Why Softmax Attention Outperforms Linear Attention

84% match2023

Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective

82% match2025

cosFormer: Rethinking Softmax in Attention

82% match1 repo1992022

Cite this paper

@article{hu2026transformer,
  title  = {Transformer Approximations from ReLUs},
  author = {Jerry Yao-Chieh Hu and Mingcheng Lu and Yi-Chen Lee and Han Liu},
  year   = {2026},
  eprint = {2604.24878},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2604.24878v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2604.24878.svg)](https://codeofpaper.com/paper/2604.24878)

Discussion