Reinforced Self-Training (ReST) for Language Modeling

Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts +9 more
8/17/2023
cs.CLcs.LG

No official code found. Best implementation: kyegomez/ReST (6 stars, MIT, last commit 2023-08-29).

Abstract

Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a simple algorithm for aligning LLMs with human preferences inspired by growing batch reinforcement learning (RL), which we call Reinforced Self-Training (ReST). Given an initial LLM policy, ReST produces a dataset by generating samples from the policy, which are then used to improve the LLM policy using offline RL algorithms. ReST is more efficient than typical online RLHF methods because the training dataset is produced offline, which allows data reuse. While ReST is a general approach applicable to all generative learning settings, we focus on its application to machine translation. Our results show that ReST can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.

View on arXivView PDF

Code Implementations(1)

kyegomez/ReSTCommunity59%
MIT

My implementation of "Reinforced Self-Training (ReST) for Language Modeling"

60Aug 29, 20233 years agoMIT

Similar Papers

Aligning Neural Machine Translation Models: Human Feedback in Training and Inference

92% match2023

Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution

90% match2024

SLiC-HF: Sequence Likelihood Calibration with Human Feedback

90% match1 repo52023

Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language

90% match2023

RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation

89% match1 repo2025

CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment

89% match1 repo22024

Cite this paper

@article{gulcehre2023reinforced,
  title  = {Reinforced Self-Training (ReST) for Language Modeling},
  author = {Caglar Gulcehre and Tom Le Paine and Srivatsan Srinivasan and Ksenia Konyushkova and Lotte Weerts and Abhishek Sharma and Aditya Siddhant and Alex Ahern and Miaosen Wang and Chenjie Gu and Wolfgang Macherey and Arnaud Doucet and Orhan Firat and Nando de Freitas},
  year   = {2023},
  eprint = {2308.08998},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.08998v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.08998.svg)](https://codeofpaper.com/paper/2308.08998)

Discussion