Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup

Tristan Kenneweg, Philip Kenneweg, Barbara Hammer
2/26/2024
cs.IRcs.CL

No official code found. Best implementation: TKenneweg/RAG_Dataset_Gen (3 stars, last commit 2024-06-20).

Abstract

Retrieval Augmented Generation (RAG) systems have seen huge popularity in augmenting Large-Language Model (LLM) outputs with domain specific and time sensitive data. Very recently a shift is happening from simple RAG setups that query a vector database for additional information with every user input to more sophisticated forms of RAG. However, different concrete approaches compete on mostly anecdotal evidence at the moment. In this paper we present a rigorous dataset creation and evaluation workflow to quantitatively compare different RAG strategies. We use a dataset created this way for the development and evaluation of a boolean agent RAG setup: A system in which a LLM can decide whether to query a vector database or not, thus saving tokens on questions that can be answered with internal knowledge. We publish our code and generated dataset online.

View on arXivView PDF

Code Implementations(1)

TKenneweg/RAG_Dataset_GenCommunity100%
30PythonFeb 26, 20242 years ago

Similar Papers

Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems

88% match2024

A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions

88% match2024

RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation

87% match1 repo7712024

Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely

87% match2024

Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey

87% match2025

Deploying Large Language Models With Retrieval Augmented Generation

87% match1 repo2024

Cite this paper

@article{kenneweg2024retrieval,
  title  = {Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup},
  author = {Tristan Kenneweg and Philip Kenneweg and Barbara Hammer},
  year   = {2024},
  eprint = {2403.00820},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2403.00820v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2403.00820.svg)](https://codeofpaper.com/paper/2403.00820)

Discussion