YaRN: Efficient Context Window Extension of Large Language Models

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole
2/12/2026

No official code found. Best implementation: jquesnelle/yarn (1,793 stars, MIT, last commit 2024-04-17).

Abstract

Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models. However, these models fail to generalize past the sequence length they were trained on. We present YaRN (Yet another RoPE extensioN method), a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps than previous methods. Using YaRN, we show that LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow, while also surpassing previous the state-of-the-art at context window extension. In addition, we demonstrate that YaRN exhibits the capability to extrapolate beyond the limited context of a fine-tuning dataset. Code is available at https://github.com/jquesnelle/yarn

View on arXivView PDF

Code Implementations(1)

jquesnelle/yarnCommunity100%
MIT

YaRN: Efficient Context Window Extension of Large Language Models

1,793137Shell, PythonJun 26, 20232 years agoMIT

Similar Papers

Exploring Context Window of Large Language Models via Decomposed Positional Vectors

86% match2026

PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training

85% match2026

DAPE: Data-Adaptive Positional Encoding for Length Extrapolation

85% match1 repo412026

DAPE: Data-Adaptive Positional Encoding for Length Extrapolation

85% match1 repo412024

Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation

85% match1 repo122025

Length Generalization of Causal Transformers without Position Encoding

84% match1 repo292024

Cite this paper

@article{peng2026yarn,
  title  = {YaRN: Efficient Context Window Extension of Large Language Models},
  author = {Bowen Peng and Jeffrey Quesnelle and Honglu Fan and Enrico Shippole},
  year   = {2026},
  eprint = {2309.00071},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/pdf/2309.00071},
  journal = {ICLR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.00071.svg)](https://codeofpaper.com/paper/2309.00071)

Discussion