Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation

Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, P. R. Kumar +1 more
7/17/2023
cs.LGcs.ROmath.OC

Official code: tliu1997/RNAC (12 stars, last commit 2024-10-20). 1 more implementation below.

Abstract

We study robust reinforcement learning (RL) with the goal of determining a well-performing policy that is robust against model mismatch between the training simulator and the testing environment. Previous policy-based robust RL algorithms mainly focus on the tabular setting under uncertainty sets that facilitate robust policy evaluation, but are no longer tractable when the number of states scales up. To this end, we propose two novel uncertainty set formulations, one based on double sampling and the other on an integral probability metric. Both make large-scale robust RL tractable even when one only has access to a simulator. We propose a robust natural actor-critic (RNAC) approach that incorporates the new uncertainty sets and employs function approximation. We provide finite-time convergence guarantees for the proposed RNAC algorithm to the optimal robust policy within the function approximation error. Finally, we demonstrate the robust performance of the policy learned by our proposed RNAC approach in multiple MuJoCo environments and a real-world TurtleBot navigation task.

View on arXivView PDF

Code Implementations(2)

tliu1997/RNACOfficial64%

[NeurIPS 2023] "Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation"

122PythonOct 12, 20232 years ago
arjunmahesh1/RNACCommunity63%

[NeurIPS 2023] "Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation"

00PythonJan 5, 20262 years ago

Similar Papers

On Practical Robust Reinforcement Learning: Practical Uncertainty Set and Double-Agent Algorithm

86% match2023

Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Model

86% match1 repo32024

DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty

86% match2 repos62025

Efficient Q-Learning and Actor-Critic Methods for Robust Average Reward Reinforcement Learning

85% match2025

Robust Reinforcement Learning with Dynamic Distortion Risk Measures

85% match1 repo12024

Efficient Action Robust Reinforcement Learning with Probabilistic Policy Execution Uncertainty

85% match2023

Cite this paper

@article{zhou2023natural,
  title  = {Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation},
  author = {Ruida Zhou and Tao Liu and Min Cheng and Dileep Kalathil and P. R. Kumar and Chao Tian},
  year   = {2023},
  eprint = {2307.08875},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2307.08875v2},
  journal = {NEURIPS 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2307.08875.svg)](https://codeofpaper.com/paper/2307.08875)

Discussion