A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems

Florin Cuconasu, Giovanni Trappolini, Nicola Tonellotto, Fabrizio Silvestri
6/21/2024
cs.CLcs.IR

No official code found. Best implementation: florin-git/Base-vs-Instruct-LLMs-in-RAG-Systems (8 stars, GPL-3.0, last commit 2024-07-03).

Abstract

Retrieval Augmented Generation (RAG) represents a significant advancement in artificial intelligence combining a retrieval phase with a generative phase, with the latter typically being powered by large language models (LLMs). The current common practices in RAG involve using "instructed" LLMs, which are fine-tuned with supervised training to enhance their ability to follow instructions and are aligned with human preferences using state-of-the-art techniques. Contrary to popular belief, our study demonstrates that base models outperform their instructed counterparts in RAG tasks by 20% on average under our experimental settings. This finding challenges the prevailing assumptions about the superiority of instructed LLMs in RAG applications. Further investigations reveal a more nuanced situation, questioning fundamental aspects of RAG and suggesting the need for broader discussions on the topic; or, as Fromm would have it, "Seldom is a glance at the statistics enough to understand the meaning of the figures".

View on arXivView PDF

Code Implementations(1)

florin-git/Base-vs-Instruct-LLMs-in-RAG-SystemsCommunity100%
GPL-3.0

Code and data for "A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems"

80Shell, PythonJun 20, 20242 years agoGPL-3.0

Similar Papers

Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models

87% match2024

LLMs are Biased Evaluators But Not Biased for Retrieval Augmented Generation

86% match1 repo12024

Post-training an LLM for RAG? Train on Self-Generated Demonstrations

86% match2025

A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models

86% match2024

Trustworthiness in Retrieval-Augmented Generation Systems: A Survey

86% match1 repo162024

RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation

86% match1 repo12025

Cite this paper

@article{cuconasu2024tale,
  title  = {A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems},
  author = {Florin Cuconasu and Giovanni Trappolini and Nicola Tonellotto and Fabrizio Silvestri},
  year   = {2024},
  eprint = {2406.14972},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.14972v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.14972.svg)](https://codeofpaper.com/paper/2406.14972)

Discussion