Contextual Position Encoding: Learning to Count What's Important

Olga Golovneva, Tianlu Wang, Jason Weston, Sainbayar Sukhbaatar
5/29/2024
cs.CLcs.AI

No official code found. Best implementation: juvi21/CoPE-cuda (22 stars, last commit 2024-06-05). 1 more below.

Abstract

The attention mechanism is a critical component of Large Language Models (LLMs) that allows tokens in a sequence to interact with each other, but is order-invariant. Incorporating position encoding (PE) makes it possible to address by position, such as attending to the i-th token. However, current PE methods use token counts to derive position, and thus cannot generalize to higher levels of abstraction, such as attending to the i-th sentence. In this paper, we propose a new position encoding method, Contextual Position Encoding (CoPE), that allows positions to be conditioned on context by incrementing position only on certain tokens determined by the model. This allows more general position addressing such as attending to the $i$-th particular word, noun, or sentence. We show that CoPE can solve the selective copy, counting and Flip-Flop tasks where popular position embeddings fail, and improves perplexity on language modeling and coding tasks.

View on arXivView PDF

Code Implementations(2)

QWTforGithub/The-code-interpretation-of-CoPECommunity64%

The code interpretation of the paper : Contextual Position Encoding: Learning to Count What’s Important

31PythonJul 3, 20242 years ago
juvi21/CoPE-cudaCommunity63%

Contextual Position Encoding but with some custom CUDA Kernels https://arxiv.org/abs/2405.18719

221C++, CudaJun 4, 20242 years ago

Similar Papers

Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding

85% match1 repo152025

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

85% match1 repo352026

Attention Instruction: Amplifying Attention in the Middle via Prompting

84% match1 repo52024

Layer-wise Positional Bias in Short-Context Language Modeling

84% match2026

When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training

84% match1 repo2162024

Eliminating Position Bias of Language Models: A Mechanistic Approach

83% match1 repo232026

Cite this paper

@article{golovneva2024contextual,
  title  = {Contextual Position Encoding: Learning to Count What's Important},
  author = {Olga Golovneva and Tianlu Wang and Jason Weston and Sainbayar Sukhbaatar},
  year   = {2024},
  eprint = {2405.18719},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2405.18719v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.18719.svg)](https://codeofpaper.com/paper/2405.18719)

Discussion