Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control

Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, Marek Cygan
5/25/2024
cs.LG

No official code found. Best implementation: naumix/BiggerRegularizedOptimistic (61 stars, MIT, last commit 2025-01-29).

Abstract

Sample efficiency in Reinforcement Learning (RL) has traditionally been driven by algorithmic enhancements. In this work, we demonstrate that scaling can also lead to substantial improvements. We conduct a thorough investigation into the interplay of scaling model capacity and domain-specific RL enhancements. These empirical findings inform the design choices underlying our proposed BRO (Bigger, Regularized, Optimistic) algorithm. The key innovation behind BRO is that strong regularization allows for effective scaling of the critic networks, which, paired with optimistic exploration, leads to superior performance. BRO achieves state-of-the-art results, significantly outperforming the leading model-based and model-free algorithms across 40 complex tasks from the DeepMind Control, MetaWorld, and MyoSuite benchmarks. BRO is the first model-free algorithm to achieve near-optimal policies in the notoriously challenging Dog and Humanoid tasks.

View on arXivView PDF

Code Implementations(1)

naumix/BiggerRegularizedOptimisticCommunity100%
MIT

Official implementation of the BRO algorithm

618Shell, PythonJun 25, 20241 years agoMIT

Similar Papers

Each Prompt Matters: Scaling Reinforcement Learning Without Wasting Rollouts on Hundred-Billion-Scale MoE

87% match2025

BroRL: Scaling Reinforcement Learning via Broadened Exploration

87% match2025

The Art of Scaling Reinforcement Learning Compute for LLMs

86% match2025

Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization

86% match2025

Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training

85% match2025

Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model

85% match2025

Cite this paper

@article{nauman2024bigger,
  title  = {Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control},
  author = {Michal Nauman and Mateusz Ostaszewski and Krzysztof Jankowski and Piotr Miłoś and Marek Cygan},
  year   = {2024},
  eprint = {2405.16158},
  archivePrefix = {arXiv},
  doi    = {10.48550/arXiv.2405.16158},
  url    = {http://arxiv.org/abs/2405.16158v3},
  journal = {NEURIPS 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.16158.svg)](https://codeofpaper.com/paper/2405.16158)

Discussion