Robust Offline Reinforcement learning with Heavy-Tailed Rewards

Jin Zhu, Runzhe Wan, Zhengling Qi, Shikai Luo, Chengchun Shi
10/28/2023
cs.LGcs.AIstat.ML

Official code: Mamba413/ROOM (4 stars, last commit 2024-08-02).

Abstract

This paper endeavors to augment the robustness of offline reinforcement learning (RL) in scenarios laden with heavy-tailed rewards, a prevalent circumstance in real-world applications. We propose two algorithmic frameworks, ROAM and ROOM, for robust off-policy evaluation and offline policy optimization (OPO), respectively. Central to our frameworks is the strategic incorporation of the median-of-means method with offline RL, enabling straightforward uncertainty estimation for the value function estimator. This not only adheres to the principle of pessimism in OPO but also adeptly manages heavy-tailed rewards. Theoretical results and extensive experiments demonstrate that our two frameworks outperform existing methods on the logged dataset exhibits heavy-tailed reward distributions. The implementation of the proposal is available at https://github.com/Mamba413/ROOM.

View on arXivView PDF

Code Implementations(1)

Mamba413/ROOMOfficial100%

Robust Offline Reinforcement Learning with Heavy-Tailed Rewards

41Shell, PythonFeb 3, 20242 years ago
heavy-tailed-distributionsoff-policy-evaluationoffline-reinforcement-learningrobust-statistics

Similar Papers

Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning

86% match1 repo12024

Semi-pessimistic Reinforcement Learning

86% match2025

Offline Reinforcement Learning with Imputed Rewards

86% match2024

Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning

86% match2024

Optimistic Model Rollouts for Pessimistic Offline Policy Optimization

86% match2024

Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning

85% match2023

Cite this paper

@article{zhu2023robust,
  title  = {Robust Offline Reinforcement learning with Heavy-Tailed Rewards},
  author = {Jin Zhu and Runzhe Wan and Zhengling Qi and Shikai Luo and Chengchun Shi},
  year   = {2023},
  eprint = {2310.18715},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.18715v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.18715.svg)](https://codeofpaper.com/paper/2310.18715)

Discussion