Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In

Zichun Yu, Chenyan Xiong, Shi Yu, Zhiyuan Liu
5/27/2023
cs.CLcs.LG

Official code: OpenMatch/Augmentation-Adapted-Retriever (60 stars, MIT, last commit 2024-07-12).

Abstract

Retrieval augmentation can aid language models (LMs) in knowledge-intensive tasks by supplying them with external information. Prior works on retrieval augmentation usually jointly fine-tune the retriever and the LM, making them closely coupled. In this paper, we explore the scheme of generic retrieval plug-in: the retriever is to assist target LMs that may not be known beforehand or are unable to be fine-tuned together. To retrieve useful documents for unseen target LMs, we propose augmentation-adapted retriever (AAR), which learns LM's preferences obtained from a known source LM. Experiments on the MMLU and PopQA datasets demonstrate that our AAR trained with a small source LM is able to significantly improve the zero-shot generalization of larger target LMs ranging from 250M Flan-T5 to 175B InstructGPT. Further analysis indicates that the preferences of different LMs overlap, enabling AAR trained with a single source LM to serve as a generic plug-in for various target LMs. Our code is open-sourced at https://github.com/OpenMatch/Augmentation-Adapted-Retriever.

View on arXivView PDF

Code Implementations(1)

OpenMatch/Augmentation-Adapted-RetrieverOfficial100%
MIT

[ACL 2023] This is the code repo for our ACL'23 paper "Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In".

605Shell, PythonFeb 8, 20232 years agoMIT

Similar Papers

A + B: A General Generator-Reader Framework for Optimizing LLMs to Unleash Synergy Potential

84% match2024

R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generation

84% match1 repo412024

Search-Adaptor: Embedding Customization for Information Retrieval

83% match2023

Reliable, Adaptable, and Attributable Language Models with Retrieval

83% match2024

Multi-task retriever fine-tuning for domain-specific and efficient RAG

83% match2025

When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs

82% match2025

Cite this paper

@article{yu2023augmentationadapted,
  title  = {Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In},
  author = {Zichun Yu and Chenyan Xiong and Shi Yu and Zhiyuan Liu},
  year   = {2023},
  eprint = {2305.17331},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.17331v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.17331.svg)](https://codeofpaper.com/paper/2305.17331)

Discussion