Understanding the Dark Side of LLMs' Intrinsic Self-Correction

Qingjie Zhang, Di Wang, Haoting Qian, Yiming Li, Tianwei Zhang +5 more
12/19/2024
cs.CL

No official code found. Best implementation: qingjiesjtu/USC (61 stars, Apache-2.0, last commit 2024-12-20).

Abstract

Intrinsic self-correction was proposed to improve LLMs' responses via feedback prompts solely based on their inherent capability. However, recent works show that LLMs' intrinsic self-correction fails without oracle labels as feedback prompts. In this paper, we aim to interpret LLMs' intrinsic self-correction for different tasks, especially for those failure cases. By including one simple task and three complex tasks with state-of-the-art (SOTA) LLMs like ChatGPT families (o1, 4o, 3.5-turbo) and Llama families (2-7B, 3-8B, and 3.1-8B), we design three interpretation methods to reveal the dark side of LLMs' intrinsic self-correction. We identify intrinsic self-correction can (1) cause LLMs to waver both intermedia and final answers and lead to prompt bias on simple factual questions; (2) introduce human-like cognitive bias on complex tasks. In light of our findings, we also provide two simple yet effective strategies for alleviation: question repeating and supervised fine-tuning with a few samples. We open-source our work at https://x-isc.info/.

View on arXivView PDF

Code Implementations(1)

qingjiesjtu/USCCommunity61%
Apache-2.0

This is the code repository of our submission: Understanding the Dark Side of LLMs’ Intrinsic Self-Correction.

611Python, Jupyter NotebookDec 13, 20241 years agoApache-2.0

Similar Papers

Intrinsic Self-Correction in LLMs: Towards Explainable Prompting via Mechanistic Interpretability

87% match2025

On the Intrinsic Self-Correction Capability of LLMs: Uncertainty and Latent Concept

86% match1 repo22024

When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs

86% match2024

A Theoretical Understanding of Self-Correction through In-context Alignment

85% match2024

Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs

85% match1 repo82024

Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs

85% match2025

Cite this paper

@article{zhang2024understanding,
  title  = {Understanding the Dark Side of LLMs' Intrinsic Self-Correction},
  author = {Qingjie Zhang and Di Wang and Haoting Qian and Yiming Li and Tianwei Zhang and Minlie Huang and Ke Xu and Hewu Li and Yan Liu and Han Qiu},
  year   = {2024},
  eprint = {2412.14959},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.14959v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.14959.svg)](https://codeofpaper.com/paper/2412.14959)

Discussion