Model-Free Robust Average-Reward Reinforcement Learning

Yue Wang, Alvaro Velasquez, George Atia, Ashley Prater-Bennette, Shaofeng Zou
5/17/2023
cs.LGcs.AI

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

Robust Markov decision processes (MDPs) address the challenge of model uncertainty by optimizing the worst-case performance over an uncertainty set of MDPs. In this paper, we focus on the robust average-reward MDPs under the model-free setting. We first theoretically characterize the structure of solutions to the robust average-reward Bellman equation, which is essential for our later convergence analysis. We then design two model-free algorithms, robust relative value iteration (RVI) TD and robust RVI Q-learning, and theoretically prove their convergence to the optimal solution. We provide several widely used uncertainty sets as examples, including those defined by the contamination model, total variation, Chi-squared divergence, Kullback-Leibler (KL) divergence and Wasserstein distance.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Robust Average-Reward Markov Decision Processes

91% match2023

Efficient Q-Learning and Actor-Critic Methods for Robust Average Reward Reinforcement Learning

88% match2025

Provably Sample-Efficient Robust Reinforcement Learning with Average Reward

88% match2025

Bellman Optimality of Average-Reward Robust Markov Decision Processes with a Constant Gain

88% match2025

Sharper Model-free Reinforcement Learning for Average-reward Markov Decision Processes

87% match2023

On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes

87% match2024

Cite this paper

@article{wang2023modelfree,
  title  = {Model-Free Robust Average-Reward Reinforcement Learning},
  author = {Yue Wang and Alvaro Velasquez and George Atia and Ashley Prater-Bennette and Shaofeng Zou},
  year   = {2023},
  eprint = {2305.10504},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.10504v1},
  journal = {ICML 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.10504.svg)](https://codeofpaper.com/paper/2305.10504)

Discussion