Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework

Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, Lidong Bing
5/5/2023
cs.CL

No official code found. Best implementation: RuochenZhao/Verify-and-Edit (51 stars, last commit 2023-12-09).

Abstract

As large language models (LLMs) have become the norm in NLP, demonstrating good performance in generation and reasoning tasks, one of its most fatal disadvantages is the lack of factual correctness. Generating unfactual texts not only leads to lower performances but also degrades the trust and validity of their applications. Chain-of-Thought (CoT) prompting improves trust and model performance on complex reasoning tasks by generating interpretable reasoning chains, but still suffers from factuality concerns in knowledge-intensive tasks. In this paper, we propose the Verify-and-Edit framework for CoT prompting, which seeks to increase prediction factuality by post-editing reasoning chains according to external knowledge. Building on top of GPT-3, our framework lead to accuracy improvements in multiple open-domain question-answering tasks.

View on arXivView PDF

Code Implementations(1)

RuochenZhao/Verify-and-EditCommunity100%

A framework for editing the CoTs for better factuality

515PythonJan 12, 20232 years ago

Similar Papers

ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting

88% match2 repos312024

CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification

88% match1 repo422023

Stress Testing Chain-of-Thought Prompting for Large Language Models

87% match2023

Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement

87% match1 repo332023

Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation

86% match2024

Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models

86% match1 repo7352023

Cite this paper

@article{zhao2023verifyandedit,
  title  = {Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework},
  author = {Ruochen Zhao and Xingxuan Li and Shafiq Joty and Chengwei Qin and Lidong Bing},
  year   = {2023},
  eprint = {2305.03268},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.03268v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.03268.svg)](https://codeofpaper.com/paper/2305.03268)

Discussion