Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers

Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii +8 more
11/15/2023
cs.CL

Official code: yuxiaw/Factcheck-GPT (117 stars, Apache-2.0, last commit 2024-01-06).

Abstract

The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. In this work, we present a holistic end-to-end solution for annotating the factuality of LLM-generated responses, which encompasses a multi-stage annotation scheme designed to yield detailed labels concerning the verifiability and factual inconsistencies found in LLM outputs. We further construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document, aiming to facilitate the evaluation of automatic fact-checking systems. Preliminary experiments show that FacTool, FactScore and Perplexity.ai are struggling to identify false claims, with the best F1=0.63 by this annotation solution based on GPT-4. Annotation tool, benchmark and code are available at https://github.com/yuxiaw/Factcheck-GPT.

View on arXivView PDF

Code Implementations(1)

yuxiaw/Factcheck-GPTOfficial100%
Apache-2.0

Fact-Checking the Output of Generative Large Language Models in both Annotation and Evaluation.

11714HTML, PythonNov 9, 20232 years agoApache-2.0

Similar Papers

ARAFA: An LLM-Generated Arabic Fact-Checking Dataset

89% match2026

Generating Benchmarks for Factuality Evaluation of Language Models

89% match1 repo542023

FELM: Benchmarking Factuality Evaluation of Large Language Models

88% match1 repo642026

FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation

88% match2024

FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios

88% match1 repo9352023

LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond

88% match1 repo602023

Cite this paper

@article{wang2023factcheckbench,
  title  = {Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers},
  author = {Yuxia Wang and Revanth Gangi Reddy and Zain Muhammad Mujahid and Arnav Arora and Aleksandr Rubashevskii and Jiahui Geng and Osama Mohammed Afzal and Liangming Pan and Nadav Borenstein and Aditya Pillai and Isabelle Augenstein and Iryna Gurevych and Preslav Nakov},
  year   = {2023},
  eprint = {2311.09000},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.09000v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.09000.svg)](https://codeofpaper.com/paper/2311.09000)

Discussion