Formal Algorithms for Transformers

Mary Phuong, Marcus Hutter
7/19/2022
cs.LGcs.AIcs.CL

No official code found. Best implementation: myazdani/formal-algorithms-for-transformers (13 stars, last commit 2023-05-03).

Abstract

This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs.

View on arXivView PDF

Code Implementations(1)

myazdani/formal-algorithms-for-transformersCommunity67%

PyTorch implementation of algorithms in https://arxiv.org/abs/2207.09238

132Python, Jupyter NotebookMar 1, 20233 years ago

Similar Papers

An Introduction to Transformers

87% match2023

AlgoFormer: An Efficient Transformer Framework with Algorithmic Structures

85% match1 repo12024

Transformadores: Fundamentos teoricos y Aplicaciones

84% match2023

DDCL-INCRT: A Self-Organising Transformer with Hierarchical Prototype Structure (Theoretical Foundations)

84% match2026

Fully Interpretable Minimal Transformers: From Geometry to Algorithm

83% match2026

What Formal Languages Can Transformers Express? A Survey

83% match2023

Cite this paper

@article{phuong2022formal,
  title  = {Formal Algorithms for Transformers},
  author = {Mary Phuong and Marcus Hutter},
  year   = {2022},
  eprint = {2207.09238},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2207.09238v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2207.09238.svg)](https://codeofpaper.com/paper/2207.09238)

Discussion