Looped Transformers as Programmable Computers

Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D. Lee +1 more
1/30/2023
cs.LGcs.AI

No official code found. Best implementation: jysohn1108/Looped-Transformer (42 stars, last commit 2023-04-08).

Abstract

We present a framework for using transformer networks as universal computers by programming them with specific weights and placing them in a loop. Our input sequence acts as a punchcard, consisting of instructions and memory for data read/writes. We demonstrate that a constant number of encoder layers can emulate basic computing blocks, including embedding edit operations, non-linear functions, function calls, program counters, and conditional branches. Using these building blocks, we emulate a small instruction-set computer. This allows us to map iterative algorithms to programs that can be executed by a looped, 13-layer transformer. We show how this transformer, instructed by its input, can emulate a basic calculator, a basic linear algebra library, and in-context learning algorithms that employ backpropagation. Our work highlights the versatility of the attention mechanism, and demonstrates that even shallow transformers can execute full-fledged, general-purpose programs.

View on arXivView PDF

Code Implementations(1)

jysohn1108/Looped-TransformerCommunity58%

Official implementation of the transformer (TF) architecture suggested in a paper entitled "Looped Transformers as Programmable Computers"

427Shell, PythonMar 12, 20233 years ago

Similar Papers

On the Existence of Universal Simulators of Attention

85% match2025

Mechanisms of Symbol Processing for In-Context Learning in Transformer Networks

84% match2024

AlgoFormer: An Efficient Transformer Framework with Algorithmic Structures

84% match1 repo12024

Learning Transformer Programs

83% match1 repo1672023

Tighter Bounds on the Expressivity of Transformer Encoders

83% match2023

Weights to Code: Extracting Interpretable Algorithms from the Discrete Transformer

83% match2026

Cite this paper

@article{giannou2023looped,
  title  = {Looped Transformers as Programmable Computers},
  author = {Angeliki Giannou and Shashank Rajput and Jy-yong Sohn and Kangwook Lee and Jason D. Lee and Dimitris Papailiopoulos},
  year   = {2023},
  eprint = {2301.13196},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2301.13196v1},
  journal = {ICML 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2301.13196.svg)](https://codeofpaper.com/paper/2301.13196)

Discussion