Adversarially Trained Actor Critic for Offline Reinforcement Learning

Ching-An Cheng, Tengyang Xie, Nan Jiang, Alekh Agarwal
2/5/2022
cs.LG

Official code: microsoft/ATAC (74 stars, MIT, last commit 2023-02-02).

Abstract

We propose Adversarially Trained Actor Critic (ATAC), a new model-free algorithm for offline reinforcement learning (RL) under insufficient data coverage, based on the concept of relative pessimism. ATAC is designed as a two-player Stackelberg game: A policy actor competes against an adversarially trained value critic, who finds data-consistent scenarios where the actor is inferior to the data-collection behavior policy. We prove that, when the actor attains no regret in the two-player game, running ATAC produces a policy that provably 1) outperforms the behavior policy over a wide range of hyperparameters that control the degree of pessimism, and 2) competes with the best policy covered by data with appropriately chosen hyperparameters. Compared with existing works, notably our framework offers both theoretical guarantees for general function approximation and a deep RL implementation scalable to complex environments and large datasets. In the D4RL benchmark, ATAC consistently outperforms state-of-the-art offline RL algorithms on a range of continuous control tasks.

View on arXivView PDF

Code Implementations(1)

microsoft/ATACOfficial100%
MIT

Code accompanying the paper Adversarially Trained Actor Critic for Offline Reinforcement Learning by Ching-An Cheng*, Tengyang Xie*, Nan Jiang, and Alekh Agarwal.

747Shell, PythonFeb 2, 20223 years agoMIT

Similar Papers

Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief

91% match1 repo2.8k2022

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning

91% match2023

MOPO: Model-based Offline Policy Optimization

88% match1 repo1902020

Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage

87% match2023

Online Learning with Off-Policy Feedback

87% match2022

Generalization in offline RL: The structure is more important than the amount of pessimism

87% match2026

Cite this paper

@article{cheng2022adversarially,
  title  = {Adversarially Trained Actor Critic for Offline Reinforcement Learning},
  author = {Ching-An Cheng and Tengyang Xie and Nan Jiang and Alekh Agarwal},
  year   = {2022},
  eprint = {2202.02446},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2202.02446}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2202.02446.svg)](https://codeofpaper.com/paper/2202.02446)

Discussion