Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification

Thomas Kwa, Drake Thomas, Adrià Garriga-Alonso
7/19/2024
cs.LG

No official code found. Best implementation: tkwa/catastrophic-goodhart (0 stars, last commit 2024-10-25).

Abstract

When applying reinforcement learning from human feedback (RLHF), the reward is learned from data and, therefore, always has some error. It is common to mitigate this by regularizing the policy with KL divergence from a base model, with the hope that balancing reward with regularization will achieve desirable outcomes despite this reward misspecification. We show that when the reward function has light-tailed error, optimal policies under less restrictive KL penalties achieve arbitrarily high utility. However, if error is heavy-tailed, some policies obtain arbitrarily high reward despite achieving no more utility than the base model--a phenomenon we call catastrophic Goodhart. We adapt a discrete optimization method to measure the tails of reward models, finding that they are consistent with light-tailed error. However, the pervasiveness of heavy-tailed distributions in many real-world applications indicates that future sources of RL reward could have heavy-tailed error, increasing the likelihood of reward hacking even with KL regularization.

View on arXivView PDF

Code Implementations(1)

tkwa/catastrophic-goodhartCommunity100%

Plots and empirical results for Catastrophic Goodhart https://www.lesswrong.com/s/6rhjdbnEXoek4YiH7

00TeX, ShellApr 12, 20241 years ago

Similar Papers

Gradient Regularization Prevents Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards

90% match2026

Learning a Pessimistic Reward Model in RLHF

88% match2025

Generalisation of RLHF under Reward Shift and Clipped KL Regularisation

88% match2026

Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization

88% match2025

Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking

88% match2025

The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret

86% match2024

Cite this paper

@article{kwa2024catastrophic,
  title  = {Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification},
  author = {Thomas Kwa and Drake Thomas and Adrià Garriga-Alonso},
  year   = {2024},
  eprint = {2407.14503},
  archivePrefix = {arXiv},
  doi    = {10.48550/arXiv.2407.14503},
  url    = {http://arxiv.org/abs/2407.14503v2},
  journal = {NEURIPS 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.14503.svg)](https://codeofpaper.com/paper/2407.14503)

Discussion