Multi-objective Reinforcement learning from AI Feedback

Marcus Williams
6/11/2024
cs.LG

No official code found. Best implementation: marcus-jw/Multi-Objective-Reinforcement-Learning-from-AI-Feedback (9 stars, MIT, last commit 2024-06-12).

Abstract

This paper presents Multi-Objective Reinforcement Learning from AI Feedback (MORLAIF), a novel approach to improving the alignment and performance of language models trained using reinforcement learning from AI feedback (RLAIF). In contrast to standard approaches that train a single preference model to represent all human preferences, MORLAIF decomposes this task into multiple simpler principles, such as toxicity, factuality, and sycophancy. Separate preference models are trained for each principle using feedback from GPT-3.5-Turbo. These preference model scores are then combined using different scalarization functions to provide a reward signal for Proximal Policy Optimization (PPO) training of the target language model. Our experiments indicate that MORLAIF outperforms the standard RLAIF baselines and that MORLAIF can be used to align larger language models using smaller ones. Surprisingly, the choice of scalarization function does not appear to significantly impact the results.

View on arXivView PDF

Code Implementations(1)

marcus-jw/Multi-Objective-Reinforcement-Learning-from-AI-FeedbackCommunity100%
MIT

Implementation of a Multi-Objective Reinforcement Learning from AI Feedback system using torch, transformers and TRL. This project aims to test whether switching to a Multi-objective reward function in the preference model improves the safety-performance of the final model.

90Shell, PythonOct 22, 20232 years agoMIT

Similar Papers

Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization

88% match1 repo1012023

A Critical Evaluation of AI Feedback for Aligning Large Language Models

88% match1 repo1012024

MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment

88% match2025

Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback

88% match2026

Doubly Robust Alignment for Large Language Models

87% match1 repo142025

Multi-Objective Reinforcement Learning for Large Language Model Optimization: Visionary Perspective

87% match2025

Cite this paper

@article{williams2024multiobjective,
  title  = {Multi-objective Reinforcement learning from AI Feedback},
  author = {Marcus Williams},
  year   = {2024},
  eprint = {2406.07295},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2406.07295v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.07295.svg)](https://codeofpaper.com/paper/2406.07295)

Discussion