Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data

Sunil Madhow, Dan Qiao, Ming Yin, Yu-Xiang Wang
6/24/2023
cs.LGcs.AI

No confident code implementation found for this paper. 1 possible match can be shown below.

Abstract

Developing theoretical guarantees on the sample complexity of offline RL methods is an important step towards making data-hungry RL algorithms practically viable. Currently, most results hinge on unrealistic assumptions about the data distribution -- namely that it comprises a set of i.i.d. trajectories collected by a single logging policy. We consider a more general setting where the dataset may have been gathered adaptively. We develop theory for the TMIS Offline Policy Evaluation (OPE) estimator in this generalized setting for tabular MDPs, deriving high-probability, instance-dependent bounds on its estimation error. We also recover minimax-optimal offline learning in the adaptive setting. Finally, we conduct simulations to empirically analyze the behavior of these estimators under adaptive and non-adaptive regimes.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 1 weaker match is hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures

90% match2025

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

89% match1 repo642023

Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage

88% match2023

Evaluation-Time Policy Switching for Offline Reinforcement Learning

87% match2025

Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples

87% match2024

Sample-Efficient Policy Constraint Offline Deep Reinforcement Learning based on Sample Filtering

87% match2025

Cite this paper

@article{madhow2023offline,
  title  = {Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data},
  author = {Sunil Madhow and Dan Qiao and Ming Yin and Yu-Xiang Wang},
  year   = {2023},
  eprint = {2306.14063},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.14063v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.14063.svg)](https://codeofpaper.com/paper/2306.14063)

Discussion