RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models

Yasuto Hoshi, Daisuke Miyashita, Youyang Ng, Kento Tatsuno, Yasuhiro Morioka +2 more
8/21/2023
cs.CLcs.AI

Official code: yhoshi3/RaLLe (54 stars, MIT, last commit 2023-10-16).

Abstract

Retrieval-augmented large language models (R-LLMs) combine pre-trained large language models (LLMs) with information retrieval systems to improve the accuracy of factual question-answering. However, current libraries for building R-LLMs provide high-level abstractions without sufficient transparency for evaluating and optimizing prompts within specific inference processes such as retrieval and generation. To address this gap, we present RaLLe, an open-source framework designed to facilitate the development, evaluation, and optimization of R-LLMs for knowledge-intensive tasks. With RaLLe, developers can easily develop and evaluate R-LLMs, improving hand-crafted prompts, assessing individual inference processes, and objectively measuring overall system performance quantitatively. By leveraging these features, developers can enhance the performance and accuracy of their R-LLMs in knowledge-intensive generation tasks. We open-source our code at https://github.com/yhoshi3/RaLLe.

View on arXivView PDF

Code Implementations(1)

yhoshi3/RaLLeOfficial100%
MIT

RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models

544PythonAug 2, 20233 years agoMIT
large-language-modelsprompt-engineeringretrieval-augmented-generationretrieval-augmented-qaretrieval-based-chatbots

Similar Papers

Self-Convinced Prompting: Few-Shot Question Answering with Repeated Introspection

88% match2023

Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy

88% match2023

LLatrieval: LLM-Verified Retrieval for Verifiable Generation

87% match1 repo332023

Reimagining Retrieval Augmented Language Models for Answering Queries

87% match2023

Marathon: A Race Through the Realm of Long Context with Large Language Models

87% match1 repo102023

DeepThink3D: Enhancing Large Language Models with Programmatic Reasoning in Complex 3D Situated Reasoning Tasks

87% match2025

Cite this paper

@article{hoshi2023ralle,
  title  = {RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models},
  author = {Yasuto Hoshi and Daisuke Miyashita and Youyang Ng and Kento Tatsuno and Yasuhiro Morioka and Osamu Torii and Jun Deguchi},
  year   = {2023},
  eprint = {2308.10633},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.10633v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.10633.svg)](https://codeofpaper.com/paper/2308.10633)

Discussion