A Residual-Aware Theory of Position Bias in Transformers

Hanna Herasimchyk, Robin Labryga, Tomislav Prusina, Sören Laue
2/18/2026
cs.LG

No official code found. Best implementation: ml-uhh/position-bias (10 stars, MIT, last commit 2026-02-20).

Abstract

Transformer models systematically favor certain token positions, yet the architectural origins of this position bias remain poorly understood. Under causal masking at infinite depth, prior theoretical analyses of attention rollout predict an inevitable collapse of attention onto the first token. Such collapse, however, does not occur in practice. We resolve this discrepancy with a residual-aware theory of cumulative attention rollout. By incorporating residual connections, we show that this architectural component prevents collapse under realistic conditions. At finite depth, we prove that causal Transformers induce a U-shaped position bias, with attention concentrating on early and late tokens. This result provides a principled architectural explanation for the Lost-in-the-Middle phenomenon.

View on arXivView PDF

Code Implementations(1)

ml-uhh/position-biasCommunity63%
MIT

Code for "A Residual-Aware Theory of Position Bias in Transformers"

100Shell, PythonFeb 18, 20268 months agoMIT

Similar Papers

On the Emergence of Position Bias in Transformers

90% match1 repo112026

Breaking Symmetry When Training Transformers

86% match2024

LayerNorm Induces Recency Bias in Transformer Decoders

86% match1 repo42025

Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding

85% match2025

Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings

84% match2024

Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers

84% match2026

Cite this paper

@article{herasimchyk2026residualaware,
  title  = {A Residual-Aware Theory of Position Bias in Transformers},
  author = {Hanna Herasimchyk and Robin Labryga and Tomislav Prusina and Sören Laue},
  year   = {2026},
  eprint = {2602.16837},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2602.16837v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2602.16837.svg)](https://codeofpaper.com/paper/2602.16837)

Discussion