On the Generalization Ability of Retrieval-Enhanced Transformers

Tobias Norlund, Ehsan Doostmohammadi, Richard Johansson, Marco Kuhlmann
2/23/2023
cs.CL

Official code: TobiasNorlund/retro (47 stars, Apache-2.0, last commit 2024-06-04).

Abstract

Recent work on the Retrieval-Enhanced Transformer (RETRO) model has shown that off-loading memory from trainable weights to a retrieval database can significantly improve language modeling and match the performance of non-retrieval models that are an order of magnitude larger in size. It has been suggested that at least some of this performance gain is due to non-trivial generalization based on both model weights and retrieval. In this paper, we try to better understand the relative contributions of these two components. We find that the performance gains from retrieval largely originate from overlapping tokens between the database and the test data, suggesting less non-trivial generalization than previously assumed. More generally, our results point to the challenges of evaluating the generalization of retrieval-augmented language models such as RETRO, as even limited token overlap may significantly decrease test-time loss. We release our code and model at https://github.com/TobiasNorlund/retro

View on arXivView PDF

Code Implementations(1)

TobiasNorlund/retroOfficial100%
Apache-2.0

Official repo to On the Generalization Ability of Retrieval-Enhanced Transformers

476Shell, PythonJan 27, 20232 years agoApache-2.0

Similar Papers

Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models

88% match1 repo32023

To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining

86% match1 repo62026

When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets

85% match1 repo52023

Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories

85% match2023

On Retrieval Augmentation and the Limitations of Language Model Training

84% match2023

REPLUG: Retrieval-Augmented Black-Box Language Models

84% match2023

Cite this paper

@article{norlund2023generalization,
  title  = {On the Generalization Ability of Retrieval-Enhanced Transformers},
  author = {Tobias Norlund and Ehsan Doostmohammadi and Richard Johansson and Marco Kuhlmann},
  year   = {2023},
  eprint = {2302.12128},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2302.12128v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2302.12128.svg)](https://codeofpaper.com/paper/2302.12128)

Discussion