Parameter Efficient Reinforcement Learning from Human Feedback

Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen +14 more
3/15/2024
cs.LGcs.AIcs.CL

No confident code implementation found for this paper. 4 possible matches can be shown below.

Abstract

While Reinforcement Learning from Human Feedback (RLHF) effectively aligns pretrained Large Language and Vision-Language Models (LLMs, and VLMs) with human preferences, its computational cost and complexity hamper its wider adoption. To alleviate some of the computational burden of fine-tuning, parameter efficient methods, like LoRA were introduced. In this work, we empirically evaluate the setup of Parameter Efficient Reinforcement Learning from Human Feedback (PE-RLHF) that leverages LoRA fine-tuning for Reward Modeling, and Reinforcement Learning. We benchmark the PE-RLHF setup on six diverse datasets spanning summarization, harmless/helpful response generation, UI automation, and visual question answering in terms of effectiveness of the trained models, and the training resources required. Our findings show, for the first time, that PE-RLHF achieves comparable performance to RLHF, while significantly reducing training time (up to 90% faster for reward models, and 30% faster for RL), and memory footprint (up to 50% reduction for reward models, and 27% for RL). We provide comprehensive ablations across LoRA ranks, and model sizes for both reward modeling and reinforcement learning. By mitigating the computational burden associated with RLHF, we push for a broader adoption of PE-RLHF as an alignment technique for LLMs and VLMs.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 4 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning

91% match1 repo82025

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

91% match2024

Prototypical Reward Network for Data-Efficient RLHF

90% match2024

Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback

90% match2025

Teaching Large Language Models to Reason with Reinforcement Learning

89% match2024

Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models

89% match1 repo542025

Cite this paper

@article{sidahmed2024parameter,
  title  = {Parameter Efficient Reinforcement Learning from Human Feedback},
  author = {Hakim Sidahmed and Samrat Phatale and Alex Hutcheson and Zhuonan Lin and Zhang Chen and Zac Yu and Jarvis Jin and Simral Chaudhary and Roman Komarytsia and Christiane Ahlheim and Yonghao Zhu and Bowen Li and Saravanan Ganesh and Bill Byrne and Jessica Hoffmann and Hassan Mansoor and Wei Li and Abhinav Rastogi and Lucas Dixon},
  year   = {2024},
  eprint = {2403.10704},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2403.10704v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2403.10704.svg)](https://codeofpaper.com/paper/2403.10704)

Discussion