Transformers are Universal Predictors

Sourya Basu, Moulik Choraria, Lav R. Varshney
7/15/2023
cs.LGcs.CL

No confident code implementation found for this paper.

Abstract

We find limits to the Transformer architecture for language modeling and show it has a universal prediction property in an information-theoretic sense. We further analyze performance in non-asymptotic data regimes to understand the role of various components of the Transformer architecture, especially in the context of data-efficient training. We validate our theoretical analysis with experiments on both synthetic and real datasets.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Understanding Transformers via N-gram Statistics

86% match1 repo322024

Can Transformers Learn n-gram Language Models?

86% match2024

Transformers Can Represent n-gram Language Models

84% match2024

On the Universality of Transformer Architectures; How Much Attention Is Enough?

84% match2025

A Meta-Learning Perspective on Transformers for Causal Language Modeling

84% match2023

Dynamic Universal Approximation Theory: The Basic Theory for Transformer-based Large Language Models

84% match2024

Cite this paper

@article{basu2023transformers,
  title  = {Transformers are Universal Predictors},
  author = {Sourya Basu and Moulik Choraria and Lav R. Varshney},
  year   = {2023},
  eprint = {2307.07843},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2307.07843v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2307.07843.svg)](https://codeofpaper.com/paper/2307.07843)

Discussion