Adversarial Training of Reward Models

Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal +4 more
4/8/2025
cs.LG

No confident code implementation found for this paper. 4 possible matches can be shown below.

Abstract

Reward modeling has emerged as a promising approach for the scalable alignment of language models. However, contemporary reward models (RMs) often lack robustness, awarding high rewards to low-quality, out-of-distribution (OOD) samples. This can lead to reward hacking, where policies exploit unintended shortcuts to maximize rewards, undermining alignment. To address this challenge, we introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples -- responses that receive high rewards from the target RM but are OOD and of low quality. By leveraging reinforcement learning, Adv-RM trains a policy to generate adversarial examples that reliably expose vulnerabilities in large state-of-the-art reward models such as Nemotron 340B RM. Incorporating these adversarial examples into the reward training process improves the robustness of RMs, mitigating reward hacking and enhancing downstream performance in RLHF. We demonstrate that Adv-RM significantly outperforms conventional RM training, increasing stability and enabling more effective RLHF training in both synthetic and real-data settings.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 4 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Reward Models Can Improve Themselves: Reward-Guided Adversarial Failure Mode Discovery for Robust Reward Modeling

88% match2025

Unifying Adversarial Robustness and Training Across Text Scoring Models

88% match2026

RRM: Robust Reward Model Training Mitigates Reward Hacking

88% match2024

One Token to Fool LLM-as-a-Judge

88% match2025

Policy Disruption in Reinforcement Learning:Adversarial Attack with Large Language Models and Critical State Identification

87% match2025

Reward Stealing Attack on Large Language Models

87% match1 repo2026

Cite this paper

@article{bukharin2025adversarial,
  title  = {Adversarial Training of Reward Models},
  author = {Alexander Bukharin and Haifeng Qian and Shengyang Sun and Adithya Renduchintala and Soumye Singhal and Zhilin Wang and Oleksii Kuchaiev and Olivier Delalleau and Tuo Zhao},
  year   = {2025},
  eprint = {2504.06141},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2504.06141v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2504.06141.svg)](https://codeofpaper.com/paper/2504.06141)

Discussion