Check-COVID: Fact-Checking COVID-19 News Claims with Scientific Evidence

Gengyu Wang, Kate Harwood, Lawrence Chillrud, Amith Ananthram, Melanie Subbiah +1 more
5/29/2023
cs.CLcs.AIcs.CY

Official code: posuer/Check-COVID (7 stars, MIT, last commit 2023-05-31).

Abstract

We present a new fact-checking benchmark, Check-COVID, that requires systems to verify claims about COVID-19 from news using evidence from scientific articles. This approach to fact-checking is particularly challenging as it requires checking internet text written in everyday language against evidence from journal articles written in formal academic language. Check-COVID contains 1, 504 expert-annotated news claims about the coronavirus paired with sentence-level evidence from scientific journal articles and veracity labels. It includes both extracted (journalist-written) and composed (annotator-written) claims. Experiments using both a fact-checking specific system and GPT-3.5, which respectively achieve F1 scores of 76.99 and 69.90 on this task, reveal the difficulty of automatically fact-checking both claim types and the importance of in-domain data for good performance. Our data and models are released publicly at https://github.com/posuer/Check-COVID.

View on arXivView PDF

Code Implementations(1)

posuer/Check-COVIDOfficial100%
MIT

A COVID-19 News Fact-checking Dataset. Verifying claims in news against evidence in scientific papers.

70PythonJan 30, 20223 years agoMIT

Similar Papers

Pipeline and Dataset Generation for Automated Fact-checking in Almost Any Language

86% match1 repo42023

Scientific Fact-Checking: A Survey of Resources and Approaches

84% match2023

PANACEA: An Automated Misinformation Detection System on COVID-19

84% match2023

Automated Claim Matching with Large Language Models: Empowering Fact-Checkers in the Fight Against Misinformation

84% match1 repo32023

Towards Robust Fact-Checking: A Multi-Agent System with Advanced Evidence Retrieval

83% match1 repo142025

The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Who

83% match1 repo22023

Cite this paper

@article{wang2023checkcovid,
  title  = {Check-COVID: Fact-Checking COVID-19 News Claims with Scientific Evidence},
  author = {Gengyu Wang and Kate Harwood and Lawrence Chillrud and Amith Ananthram and Melanie Subbiah and Kathleen McKeown},
  year   = {2023},
  eprint = {2305.18265},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.18265v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.18265.svg)](https://codeofpaper.com/paper/2305.18265)

Discussion