FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions

Daniel Marta, Simon Holk, Miguel Vasco, Jens Lundell, Timon Homberger +4 more
4/14/2025
cs.ROcs.LG

No official code found. Best implementation: DanielLSM/flora (5 stars, last commit 2025-05-18).

Abstract

Preference-based reinforcement learning (PbRL) is a suitable approach for style adaptation of pre-trained robotic behavior: adapting the robot's policy to follow human user preferences while still being able to perform the original task. However, collecting preferences for the adaptation process in robotics is often challenging and time-consuming. In this work we explore the adaptation of pre-trained robots in the low-preference-data regime. We show that, in this regime, recent adaptation approaches suffer from catastrophic reward forgetting (CRF), where the updated reward model overfits to the new preferences, leading the agent to become unable to perform the original task. To mitigate CRF, we propose to enhance the original reward model with a small number of parameters (low-rank matrices) responsible for modeling the preference adaptation. Our evaluation shows that our method can efficiently and effectively adjust robotic behavior to human preferences across simulation benchmark tasks and multiple real-world robotic tasks.

View on arXivView PDF

Code Implementations(1)

DanielLSM/floraCommunity100%

Official codebase for "FLoRA: Efficient Preference-based Reinforcement Learning via Low-Rank Adaptation of Enhanced Reward Functions"

50Shell, PythonApr 10, 20251 years ago

Similar Papers

Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models

89% match2023

Residual Reward Models for Preference-based Reinforcement Learning

89% match2025

Adaptive Preference Scaling for Reinforcement Learning with Human Feedback

88% match1 repo62024

Provable Reward-Agnostic Preference-Based Reinforcement Learning

88% match2023

Data Driven Reward Initialization for Preference based Reinforcement Learning

87% match2023

Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity

87% match2024

Cite this paper

@article{marta2025flora,
  title  = {FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions},
  author = {Daniel Marta and Simon Holk and Miguel Vasco and Jens Lundell and Timon Homberger and Finn Busch and Olov Andersson and Danica Kragic and Iolanda Leite},
  year   = {2025},
  eprint = {2504.10002},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2504.10002v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2504.10002.svg)](https://codeofpaper.com/paper/2504.10002)

Discussion