Effective Theory of Transformers at Initialization

Emily Dinan, Sho Yaida, Susan Zhang
4/4/2023
cs.LGcs.CLhep-th

No confident code implementation found for this paper.

Abstract

We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks. This analysis suggests particular width scalings of initialization and training hyperparameters for these models. We then take up such suggestions, training Vision and Language Transformers in practical setups.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Universal Approximation Theorem for a Single-Layer Transformer

83% match2025

Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models

83% match1 repo112024

Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation

83% match2023

On the Convergence of Encoder-only Shallow Transformers

83% match2026

Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?

82% match2023

Equivariant Neural Functional Networks for Transformers

82% match2 repos32024

Cite this paper

@article{dinan2023effective,
  title  = {Effective Theory of Transformers at Initialization},
  author = {Emily Dinan and Sho Yaida and Susan Zhang},
  year   = {2023},
  eprint = {2304.02034},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.02034v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.02034.svg)](https://codeofpaper.com/paper/2304.02034)

Discussion