An Analysis of Multilingual FActScore

Kim Trong Vu, Michael Krumdick, Varshini Reddy, Franck Dernoncourt, Viet Dac Lai
6/20/2024
cs.CL

No confident code implementation found for this paper.

Abstract

FActScore has gained popularity as a metric to estimate the factuality of long-form texts generated by Large Language Models (LLMs) in English. However, there has not been any work in studying the behavior of FActScore in other languages. This paper studies the limitations of each component in the four-component pipeline of FActScore in the multilingual setting. We introduce a new dataset for FActScore on texts generated by strong multilingual LLMs. Our evaluation shows that LLMs exhibit distinct behaviors in both fact extraction and fact scoring tasks. No LLM produces consistent and reliable FActScore across languages with varying levels of resources. We also find that the knowledge source plays an important role in the quality of the estimated FActScore. Using Wikipedia as the knowledge source may hinder the true FActScore of long-form text due to its limited coverage in medium- and low-resource languages. We also incorporate three mitigations to our knowledge source that ultimately improve FActScore estimation across all languages.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScore

93% match2024

FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation

86% match1 repo4572023

Beyond Translation: LLM-Based Data Generation for Multilingual Fact-Checking

86% match1 repo52025

Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge

84% match2024

When Scale Meets Diversity: Evaluating Language Models on Fine-Grained Multilingual Claim Verification

83% match1 repo72025

Factuality of Large Language Models: A Survey

83% match2024

Cite this paper

@article{vu2024analysis,
  title  = {An Analysis of Multilingual FActScore},
  author = {Kim Trong Vu and Michael Krumdick and Varshini Reddy and Franck Dernoncourt and Viet Dac Lai},
  year   = {2024},
  eprint = {2406.19415},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.19415v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.19415.svg)](https://codeofpaper.com/paper/2406.19415)

Discussion