Learning from Sparse Offline Datasets via Conservative Density Estimation

Zhepeng Cen, Zuxin Liu, Zitong Wang, Yihang Yao, Henry Lam +1 more
1/16/2024
cs.LGcs.AI

No official code found. Best implementation: czp16/cde-offline-rl (3 stars, MIT, last commit 2023-09-27).

Abstract

Offline reinforcement learning (RL) offers a promising direction for learning policies from pre-collected datasets without requiring further interactions with the environment. However, existing methods struggle to handle out-of-distribution (OOD) extrapolation errors, especially in sparse reward or scarce data settings. In this paper, we propose a novel training algorithm called Conservative Density Estimation (CDE), which addresses this challenge by explicitly imposing constraints on the state-action occupancy stationary distribution. CDE overcomes the limitations of existing approaches, such as the stationary distribution correction method, by addressing the support mismatch issue in marginal importance sampling. Our method achieves state-of-the-art performance on the D4RL benchmark. Notably, CDE consistently outperforms baselines in challenging tasks with sparse rewards or insufficient data, demonstrating the advantages of our approach in addressing the extrapolation error problem in offline RL.

View on arXivView PDF

Code Implementations(1)

czp16/cde-offline-rlCommunity100%
MIT

Learning from Sparse Offline Datasets via Conservative Density Estimation (ICLR 2024)

30Shell, PythonMay 22, 20233 years agoMIT
d4rloffline-reinforcement-learningpytorch-implementationreinforcement-learning

Similar Papers

CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learning

88% match2024

Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity

86% match1 repo2025

Sparsity-based Safety Conservatism for Constrained Offline Reinforcement Learning

86% match2024

Conservative State Value Estimation for Offline Reinforcement Learning

86% match2023

Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage

85% match2023

DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning

85% match2023

Cite this paper

@article{cen2024learning,
  title  = {Learning from Sparse Offline Datasets via Conservative Density Estimation},
  author = {Zhepeng Cen and Zuxin Liu and Zitong Wang and Yihang Yao and Henry Lam and Ding Zhao},
  year   = {2024},
  eprint = {2401.08819},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.08819v2},
  journal = {ICLR 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.08819.svg)](https://codeofpaper.com/paper/2401.08819)

Discussion