No confident code implementation found for this paper.
In this note, we use the VC dimension technique to prove the first lower bound against one-layer softmax transformers with infinite precision. We do so for two tasks: function composition, considered by Peng, Narayanan, and Papadimitriou, and the SUM$_2$ task, considered by Sanford, Hsu, and Telgarsky.
No confident code match yet
We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.
No code implementations found yet.
Know of an implementation? Let us know in the comments below!
Strassen Attention, Split VC Dimension and Compositionality in Transformers
Parity, Sensitivity, and Transformers
Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?
From Small to Large: Generalization Bounds for Transformers on Variable-Size Inputs
Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers
Theoretical limitations of multi-layer Transformer
@article{kozachinskiy2024lower,
title = {Lower bounds on transformers with infinite precision},
author = {Alexander Kozachinskiy},
year = {2024},
eprint = {2412.20195},
archivePrefix = {arXiv},
url = {http://arxiv.org/abs/2412.20195v1}
}Shows whether official code is linked for this paper.
[](https://codeofpaper.com/paper/2412.20195)