AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG task

Herbert Ullrich, Tomáš Mlynář, Jan Drchal
10/15/2024
cs.CL

No official code found. Best implementation: aic-factcheck/aic_averitec (6 stars, last commit 2025-12-04).

Abstract

This paper describes our $3^{rd}$ place submission in the AVeriTeC shared task in which we attempted to address the challenge of fact-checking with evidence retrieved in the wild using a simple scheme of Retrieval-Augmented Generation (RAG) designed for the task, leveraging the predictive power of Large Language Models. We release our codebase and explain its two modules - the Retriever and the Evidence & Label generator - in detail, justifying their features such as MMR-reranking and Likert-scale confidence estimation. We evaluate our solution on AVeriTeC dev and test set and interpret the results, picking the GPT-4o as the most appropriate model for our pipeline at the time of our publication, with Llama 3.1 70B being a promising open-source alternative. We perform an empirical error analysis to see that faults in our predictions often coincide with noise in the data or ambiguous fact-checks, provoking further research and data augmentation.

View on arXivView PDF

Code Implementations(1)

aic-factcheck/aic_averitecCommunity100%
60Shell, PythonJul 1, 202410 months agoNOASSERTION
averitecfact-checkingretrieval-augmented-generation

Similar Papers

HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims

87% match1 repo142024

Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMs

84% match1 repo42025

Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs

83% match1 repo72024

Generative Large Language Models in Automated Fact-Checking: A Survey

82% match2024

FACT-GPT: Fact-Checking Augmentation via Claim Matching with LLMs

82% match2 repos72.8k2024

Surprising Efficacy of Fine-Tuned Transformers for Fact-Checking over Larger Language Models

82% match2024

Cite this paper

@article{ullrich2024system,
  title  = {AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG task},
  author = {Herbert Ullrich and Tomáš Mlynář and Jan Drchal},
  year   = {2024},
  eprint = {2410.11446},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.11446v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.11446.svg)](https://codeofpaper.com/paper/2410.11446)

Discussion