Online Pre-Training for Offline-to-Online Reinforcement Learning

Yongjae Shin, Jeonghye Kim, Whiyoung Jung, Sunghoon Hong, Deunsol Yoon +6 more
7/11/2025
cs.LG

No official code found. Best implementation: LGAI-Research/opt (9 stars, BSD-3-Clause, last commit 2025-09-22).

Abstract

Offline-to-online reinforcement learning (RL) aims to integrate the complementary strengths of offline and online RL by pre-training an agent offline and subsequently fine-tuning it through online interactions. However, recent studies reveal that offline pre-trained agents often underperform during online fine-tuning due to inaccurate value estimation caused by distribution shift, with random initialization proving more effective in certain cases. In this work, we propose a novel method, Online Pre-Training for Offline-to-Online RL (OPT), explicitly designed to address the issue of inaccurate value estimation in offline pre-trained agents. OPT introduces a new learning phase, Online Pre-Training, which allows the training of a new value function tailored specifically for effective online fine-tuning. Implementation of OPT on TD3 and SPOT demonstrates an average 30% improvement in performance across a wide range of D4RL environments, including MuJoCo, Antmaze, and Adroit.

View on arXivView PDF

Code Implementations(1)

LGAI-Research/optCommunity67%
BSD-3-Clause

Official implementation of "Online Pre-Training for Offline-to-Online Reinforcement Learning" (ICML 2025) by LG AI Research.

90PythonMay 23, 20251 years agoBSD-3-Clause

Similar Papers

The Three Regimes of Offline-to-Online Reinforcement Learning

90% match2025

Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions

89% match2023

Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL

89% match1 repo52024

Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data

88% match1 repo1502024

Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only

88% match2025

Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning

88% match2023

Cite this paper

@article{shin2025online,
  title  = {Online Pre-Training for Offline-to-Online Reinforcement Learning},
  author = {Yongjae Shin and Jeonghye Kim and Whiyoung Jung and Sunghoon Hong and Deunsol Yoon and Youngsoo Jang and Geonhyeong Kim and Jongseong Chae and Youngchul Sung and Kanghoon Lee and Woohyung Lim},
  year   = {2025},
  eprint = {2507.08387},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2507.08387v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2507.08387.svg)](https://codeofpaper.com/paper/2507.08387)

Discussion