A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms

Cédric Colas, Olivier Sigaud, Pierre-Yves Oudeyer
4/15/2019
stat.MEcs.LG

Official code: ccolas/rl_stats (6 stars, last commit 2019-03-04). 1 more implementation below.

Abstract

Consistently checking the statistical significance of experimental results is the first mandatory step towards reproducible science. This paper presents a hitchhiker's guide to rigorous comparisons of reinforcement learning algorithms. After introducing the concepts of statistical testing, we review the relevant statistical tests and compare them empirically in terms of false positive rate and statistical power as a function of the sample size (number of seeds) and effect size. We further investigate the robustness of these tests to violations of the most common hypotheses (normal distributions, same distributions, equal variances). Beside simulations, we compare empirical distributions obtained by running Soft-Actor Critic and Twin-Delayed Deep Deterministic Policy Gradient on Half-Cheetah. We conclude by providing guidelines and code to perform rigorous comparisons of RL algorithm performances.

View on arXivView PDF

Code Implementations(2)

ccolas/rl_statsOfficial100%

Small library to compare performance of RL algorithms

61Shell, PythonMar 4, 20197 years ago
flowersteam/rl_statsOfficial100%

Implementation of: A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms

40Shell, PythonDec 4, 20194 years ago

Similar Papers

Empirical Design in Reinforcement Learning

87% match2023

Deep Reinforcement Learning at the Edge of the Statistical Precipice

86% match1 repo8782021

Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testing

83% match2023

Statistical Inference in Reinforcement Learning: A Selective Survey

83% match1 repo52025

Beyond Expected Return: Accounting for Policy Reproducibility when Evaluating Reinforcement Learning Algorithms

83% match2023

AdaStop: adaptive statistical testing for sound comparisons of Deep RL agents

83% match2023

Cite this paper

@article{colas2019hitchhikers,
  title  = {A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms},
  author = {Cédric Colas and Olivier Sigaud and Pierre-Yves Oudeyer},
  year   = {2019},
  eprint = {1904.06979},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1904.06979}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1904.06979.svg)](https://codeofpaper.com/paper/1904.06979)

Discussion