Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO

Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos +2 more
5/25/2020
cs.LGcs.ROstat.ML
rl

Official code: MadryLab/implementation-matters (140 stars, MIT, last commit 2024-07-25).

Abstract

We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, we investigate the consequences of "code-level optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations turn out to have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function. These insights show the difficulty and importance of attributing performance gains in deep reinforcement learning. Code for reproducing our results is available at https://github.com/MadryLab/implementation-matters .

View on arXivView PDF

Code Implementations(1)

MadryLab/implementation-mattersOfficial100%
MIT
14018PythonApr 11, 20202 years agoMIT

Similar Papers

The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations

87% match1 repo442024

Identifying Policy Gradient Subspaces

85% match2024

Is Exploration or Optimization the Problem for Deep Reinforcement Learning?

85% match2025

Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers

84% match2 repos302024

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

84% match1 repo372026

Investigating the Impact of Action Representations in Policy Gradient Algorithms

84% match2023

Cite this paper

@article{engstrom2020implementation,
  title  = {Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO},
  author = {Logan Engstrom and Andrew Ilyas and Shibani Santurkar and Dimitris Tsipras and Firdaus Janoos and Larry Rudolph and Aleksander Madry},
  year   = {2020},
  eprint = {2005.12729},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2005.12729}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2005.12729.svg)](https://codeofpaper.com/paper/2005.12729)

Discussion