Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs

Víctor Gallego
2/12/2024
cs.CLcs.LG

Official code: vicgalle/refined-dpo (13 stars, MIT, last commit 2024-02-13).

Abstract

In this paper, we introduce \emph{refined Direct Preference Optimization} (rDPO), a method for improving the behavioral alignment of Large Language Models (LLMs) without the need for human-annotated data. The method involves creating synthetic data using self-critique prompting by a teacher LLM and then utilising a generalized DPO loss function to distil to a student LLM. The loss function incorporates an additional external reward model to improve the quality of synthetic data, making rDPO robust to potential noise in the synthetic dataset. rDPO is shown to be effective in a diverse set of behavioural alignment tasks, such as improved safety, robustness against role-playing, and reduced sycophancy. Code to be released at https://github.com/vicgalle/refined-dpo.

View on arXivView PDF

Code Implementations(1)

vicgalle/refined-dpoOfficial100%
MIT

Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs

131Python, Jupyter NotebookFeb 13, 20242 years agoMIT

Similar Papers

Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment

90% match1 repo142026

Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment

89% match2025

A Principled Loss Function for Direct Language Model Alignment

89% match2025

sDPO: Don't Use Your Data All at Once

89% match2024

Less is More: Improving LLM Alignment via Preference Data Selection

89% match1 repo32025

Less is More: Improving LLM Alignment via Preference Data Selection

89% match1 repo32025

Cite this paper

@article{gallego2024refined,
  title  = {Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs},
  author = {Víctor Gallego},
  year   = {2024},
  eprint = {2402.08005},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.08005v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.08005.svg)](https://codeofpaper.com/paper/2402.08005)

Discussion