A Length-Extrapolatable Transformer

Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang +4 more
12/20/2022
cs.CL

Official code: microsoft/torchscale (3,133 stars, MIT, last commit 2024-04-11).

Abstract

Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define attention resolution as an indicator of extrapolation. Then we propose two designs to improve the above metric of Transformers. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better resolution. We evaluate different Transformer variants with language modeling. Experimental results show that our model achieves strong performance in both interpolation and extrapolation settings. The code will be available at https://aka.ms/LeX-Transformer.

View on arXivView PDF

Code Implementations(1)

microsoft/torchscaleOfficial100%
MIT

Foundation Architecture for (M)LLMs

3,133225PythonNov 17, 20222 years agoMIT
computer-visionmachine-learningmultimodalnatural-language-processingpretrained-language-model+3 more

Similar Papers

Giraffe: Adventures in Expanding Context Lengths in LLMs

89% match1 repo6032023

Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding

88% match2023

Towards Infinite Length Extrapolation: A Unified Approach

86% match2026

On the token distance modeling ability of higher RoPE attention dimension

86% match2024

Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models

86% match2025

Self-Selected Attention Span for Accelerating Large Language Model Inference

86% match2024

Cite this paper

@article{sun2022lengthextrapolatable,
  title  = {A Length-Extrapolatable Transformer},
  author = {Yutao Sun and Li Dong and Barun Patra and Shuming Ma and Shaohan Huang and Alon Benhaim and Vishrav Chaudhary and Xia Song and Furu Wei},
  year   = {2022},
  eprint = {2212.10554},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2212.10554}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2212.10554.svg)](https://codeofpaper.com/paper/2212.10554)

Discussion