Looped Transformers are Better at Learning Learning Algorithms

Liu Yang, Kangwook Lee, Robert Nowak, Dimitris Papailiopoulos
2/12/2026

No official code found. Best implementation: Leiay/looped_transformer (47 stars, MIT, last commit 2023-12-12). 1 more below.

Abstract

Transformers have demonstrated effectiveness in in-context solving data-fitting problems from various (latent) models, as reported by Garg et al. However, the absence of an inherent iterative structure in the transformer architecture presents a challenge in emulating the iterative algorithms, which are commonly employed in traditional machine learning methods. To address this, we propose the utilization of looped transformer architecture and its associated training methodology, with the aim of incorporating iterative characteristics into the transformer architectures. Experimental results suggest that the looped transformer achieves performance comparable to the standard transformer in solving various data-fitting problems, while utilizing less than 10% of the parameter count.

View on arXivView PDF

Code Implementations(2)

Leiay/looped_transformerCommunity100%
MIT
4710Shell, PythonDec 12, 20232 years agoMIT
Vadimbuildercxx/looped_transformerCommunity58%
PyTorch

Experimental implementation of "Looped Transformers are Better at Learning Learning Algorithms" showing superior performance with 12x fewer parameters. Includes complete environment setup, pre-trained weights, and extensive experiments comparing Looped TFs vs traditional Transformers for In-Context Learning.

81Python, Jupyter NotebookJul 23, 20242 years ago
deep-learninglooped-transformersmachine-learningmodel-compressionpytorch+3 more

Similar Papers

Learning Transformer Programs

84% match1 repo1672023

Learning Spectral Methods by Transformers

84% match2025

Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?

84% match2024

Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection

84% match2023

Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training

84% match2026

AlgoFormer: An Efficient Transformer Framework with Algorithmic Structures

83% match1 repo12024

Cite this paper

@article{yang2026looped,
  title  = {Looped Transformers are Better at Learning Learning Algorithms},
  author = {Liu Yang and Kangwook Lee and Robert Nowak and Dimitris Papailiopoulos},
  year   = {2026},
  eprint = {2311.12424},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2311.12424},
  journal = {ICLR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.12424.svg)](https://codeofpaper.com/paper/2311.12424)

Discussion