NCL-UoR at SemEval-2025 Task 3: Detecting Multilingual Hallucination and Related Observable Overgeneration Text Spans with Modified RefChecker and Modified SeflCheckGPT

Jiaying Hong, Thanet Markchom, Jianfei Xu, Tong Wu, Huizhi Liang
3/2/2025
cs.CLcs.AI

No official code found. Best implementation: jianfeixu95/NCL-UoR (2 stars, last commit 2025-02-25).

Abstract

SemEval-2025 Task 3 (Mu-SHROOM) focuses on detecting hallucinations in content generated by various large language models (LLMs) across multiple languages. This task involves not only identifying the presence of hallucinations but also pinpointing their specific occurrences. To tackle this challenge, this study introduces two methods: modified RefChecker and modified SelfCheckGPT. The modified RefChecker integrates prompt-based factual verification into References, structuring them as claim-based tests rather than single external knowledge sources. The modified SelfCheckGPT incorporates external knowledge to overcome its reliance on internal knowledge. In addition, both methods' original prompt designs are enhanced to identify hallucinated words within LLM-generated texts. Experimental results demonstrate the effectiveness of the approach, achieving a high ranking on the test dataset in detecting hallucinations across various languages, with an average IoU of 0.5310 and an average COR of 0.5669.

View on arXivView PDF

Code Implementations(1)

jianfeixu95/NCL-UoRCommunity100%

SemEval 2025 Task 3: The Mu-SHROOM shared task

22Python, Jupyter NotebookDec 31, 20241 years ago

Similar Papers

SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes

92% match2 repos52025

HausaNLP at SemEval-2025 Task 3: Towards a Fine-Grained Model-Aware Hallucination Detection

92% match2025

HalluSearch at SemEval-2025 Task 3: A Search-Enhanced RAG Pipeline for Hallucination Detection

91% match2025

SLPL SHROOM at SemEval2024 Task 06: A comprehensive study on models ability to detect hallucination

90% match2024

REFIND at SemEval-2025 Task 3: Retrieval-Augmented Factuality Hallucination Detection in Large Language Models

90% match1 repo42025

UCSC at SemEval-2025 Task 3: Context, Models and Prompt Optimization for Automated Hallucination Detection in LLM Output

89% match1 repo22025

Cite this paper

@article{hong2025ncluor,
  title  = {NCL-UoR at SemEval-2025 Task 3: Detecting Multilingual Hallucination and Related Observable Overgeneration Text Spans with Modified RefChecker and Modified SeflCheckGPT},
  author = {Jiaying Hong and Thanet Markchom and Jianfei Xu and Tong Wu and Huizhi Liang},
  year   = {2025},
  eprint = {2503.01921},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2503.01921v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2503.01921.svg)](https://codeofpaper.com/paper/2503.01921)

Discussion