BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models

Qingqing Cao, Sewon Min, Yizhong Wang, Hannaneh Hajishirzi
2/12/2026

No official code found. Best implementation: csarron/BTR (16 stars, last commit 2024-03-03).

Abstract

Retrieval augmentation addresses many critical problems in large language models such as hallucination, staleness, and privacy leaks. However, running retrieval-augmented language models (LMs) is slow and difficult to scale due to processing large amounts of retrieved text. We introduce binary token representations (BTR), which use 1-bit vectors to precompute every token in passages, significantly reducing computation during inference. Despite the potential loss of accuracy, our new calibration techniques and training objectives restore performance. Combined with offline and runtime compression, this only requires 127GB of disk space for encoding 3 billion tokens in Wikipedia. Our experiments show that on five knowledge-intensive NLP tasks, BTR accelerates state-of-the-art inference by up to 4x and reduces storage by over 100x while maintaining over 95% task performance.

View on arXivView PDF

Code Implementations(1)

csarron/BTRCommunity100%
160Shell, PythonSep 29, 20232 years ago

Similar Papers

SparQ Attention: Bandwidth-Efficient LLM Inference

83% match1 repo362026

One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models

83% match1 repo262024

To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining

82% match1 repo62026

TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models

82% match1 repo252025

RA-DIT: Retrieval-Augmented Dual Instruction Tuning

82% match2026

FlashBack:Efficient Retrieval-Augmented Language Modeling for Long Context Inference

81% match2024

Cite this paper

@article{cao2026btr,
  title  = {BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models},
  author = {Qingqing Cao and Sewon Min and Yizhong Wang and Hannaneh Hajishirzi},
  year   = {2026},
  eprint = {2310.01329},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/pdf/2310.01329},
  journal = {ICLR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.01329.svg)](https://codeofpaper.com/paper/2310.01329)

Discussion