Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data

Jeonghye Kim, Yongjae Shin, Whiyoung Jung, Sunghoon Hong, Deunsol Yoon +3 more
7/11/2025
cs.LGcs.AI

No official code found. Best implementation: LGAI-Research/pars (4 stars, BSD-3-Clause, last commit 2025-09-22).

Abstract

Reinforcement learning with offline data suffers from Q-value extrapolation errors. To address this issue, we first demonstrate that linear extrapolation of the Q-function beyond the data range is particularly problematic. To mitigate this, we propose guiding the gradual decrease of Q-values outside the data range, which is achieved through reward scaling with layer normalization (RS-LN) and a penalization mechanism for infeasible actions (PA). By combining RS-LN and PA, we develop a new algorithm called PARS. We evaluate PARS across a range of tasks, demonstrating superior performance compared to state-of-the-art algorithms in both offline training and online fine-tuning on the D4RL benchmark, with notable success in the challenging AntMaze Ultra task.

View on arXivView PDF

Code Implementations(1)

LGAI-Research/parsCommunity67%
BSD-3-Clause

Official implementation of "Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data" (ICML 2025) by LG AI Research.

41Python, DockerfileMay 23, 20251 years agoBSD-3-Clause

Similar Papers

Offline Reinforcement Learning with On-Policy Q-Function Regularization

87% match2023

A Minimalist Approach to Offline Reinforcement Learning

85% match1 repo4132021

Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning

85% match2025

Exclusively Penalized Q-learning for Offline Reinforcement Learning

84% match1 repo62024

FAST-Q: Fast-track Exploration with Adversarially Balanced State Representations for Counterfactual Action Estimation in Offline Reinforcement Learning

84% match1 repo2025

Towards Robust Offline Reinforcement Learning under Diverse Data Corruption

84% match1 repo222023

Cite this paper

@article{kim2025penalizing,
  title  = {Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data},
  author = {Jeonghye Kim and Yongjae Shin and Whiyoung Jung and Sunghoon Hong and Deunsol Yoon and Youngchul Sung and Kanghoon Lee and Woohyung Lim},
  year   = {2025},
  eprint = {2507.08761},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2507.08761v2},
  journal = {ICML 2025 2025}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2507.08761.svg)](https://codeofpaper.com/paper/2507.08761)

Discussion