The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval

Ting-Rui Chiang, Dani Yogatama
2/16/2025
cs.CLcs.LG

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

The Rotary Position Embedding (RoPE) is widely used in the attention heads of many large language models (LLM). It rotates dimensions in the query and the key vectors by different angles according to their positions in the input sequence. For long context modeling, the range of positions may vary a lot, and thus RoPE rotates some dimensions by a great range of angles. We hypothesize that the wide range of rotation angles may prevent LLMs from utilizing those dimensions. To validate this hypothesis, we present a controlled experiment showing that applying RoPE causes low utility of certain dimensions. Our analyses on three LLMs also indicate that these dimensions do not help LLMs do long-context question answering.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Round and Round We Go! What makes Rotary Positional Encodings useful?

88% match2024

On the token distance modeling ability of higher RoPE attention dimension

88% match2024

Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs

87% match1 repo332025

Rotary Offset Features in Large Language Models

86% match2025

Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective

85% match2024

Rope to Nope and Back Again: A New Hybrid Attention Strategy

85% match2025

Cite this paper

@article{chiang2025rotary,
  title  = {The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval},
  author = {Ting-Rui Chiang and Dani Yogatama},
  year   = {2025},
  eprint = {2502.11276},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2502.11276v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2502.11276.svg)](https://codeofpaper.com/paper/2502.11276)

Discussion