Doubly Robust Alignment for Large Language Models

Erhan Xu, Kai Ye, Hongyi Zhou, Luhan Zhu, Francesco Quinzan +1 more
6/1/2025
cs.LGcs.AIstat.ML

Official code: DRPO4LLM/DRPO4LLM (14 stars, last commit 2025-05-21).

Abstract

This paper studies reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences. While RLHF has demonstrated promising results, many algorithms are highly sensitive to misspecifications in the underlying preference model (e.g., the Bradley-Terry model), the reference policy, or the reward function, resulting in undesirable fine-tuning. To address model misspecification, we propose a doubly robust preference optimization algorithm that remains consistent when either the preference model or the reference policy is correctly specified (without requiring both). Our proposal demonstrates superior and more robust performance than state-of-the-art algorithms, both in theory and in practice. The code is available at https://github.com/DRPO4LLM/DRPO4LLM

View on arXivView PDF

Code Implementations(1)

DRPO4LLM/DRPO4LLMOfficial100%
145PythonMay 20, 20251 years ago

Similar Papers

Investigating on RLHF methodology

93% match2024

Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning

93% match1 repo82025

On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization

93% match1 repo72024

Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model

92% match2024

Accelerated Preference Optimization for Large Language Model Alignment

92% match2024

SAIL: Self-Improving Efficient Online Alignment of Large Language Models

91% match2024

Cite this paper

@article{xu2025doubly,
  title  = {Doubly Robust Alignment for Large Language Models},
  author = {Erhan Xu and Kai Ye and Hongyi Zhou and Luhan Zhu and Francesco Quinzan and Chengchun Shi},
  year   = {2025},
  eprint = {2506.01183},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.01183v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.01183.svg)](https://codeofpaper.com/paper/2506.01183)

Discussion