Atlas: Few-shot Learning with Retrieval Augmented Language Models

Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni +5 more
8/5/2022
cs.CL

No official code found. Best implementation: facebookresearch/atlas (565 stars, last commit 2026-04-08).

Abstract

Large language models have shown impressive few-shot results on a wide range of tasks. However, when knowledge is key for such results, as is the case for tasks such as question answering and fact checking, massive parameter counts to store knowledge seem to be needed. Retrieval augmented models are known to excel at knowledge intensive tasks without the need for as many parameters, but it is unclear whether they work in few-shot settings. In this work we present Atlas, a carefully designed and pre-trained retrieval augmented language model able to learn knowledge intensive tasks with very few training examples. We perform evaluations on a wide range of tasks, including MMLU, KILT and NaturalQuestions, and study the impact of the content of the document index, showing that it can easily be updated. Notably, Atlas reaches over 42% accuracy on Natural Questions using only 64 examples, outperforming a 540B parameters model by 3% despite having 50x fewer parameters.

View on arXivView PDF

Code Implementations(1)

facebookresearch/atlasCommunity100%

Code repository for supporting the paper "Atlas Few-shot Learning with Retrieval Augmented Language Models",(https//arxiv.org/abs/2208.03299)

56574Shell, PythonNov 16, 20226 months agoNOASSERTION

Similar Papers

RA-DIT: Retrieval-Augmented Dual Instruction Tuning

88% match1 repo32023

RA-DIT: Retrieval-Augmented Dual Instruction Tuning

88% match2026

Few-shot Adaptation Works with UnpredicTable Data

86% match1 repo252022

Automated Few-shot Classification with Instruction-Finetuned Language Models

86% match1 repo62023

Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy

86% match2023

Meta-training with Demonstration Retrieval for Efficient Few-shot Learning

85% match2023

Cite this paper

@article{izacard2022atlas,
  title  = {Atlas: Few-shot Learning with Retrieval Augmented Language Models},
  author = {Gautier Izacard and Patrick Lewis and Maria Lomeli and Lucas Hosseini and Fabio Petroni and Timo Schick and Jane Dwivedi-Yu and Armand Joulin and Sebastian Riedel and Edouard Grave},
  year   = {2022},
  eprint = {2208.03299},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2208.03299v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2208.03299.svg)](https://codeofpaper.com/paper/2208.03299)

Discussion