SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores

Zhiyu Mei, Wei Fu, Jiaxuan Gao, Guangju Wang, Huanchen Zhang +1 more
6/29/2023
cs.DCcs.AIcs.LG

Official code: openpsi-project/srl (38 stars, last commit 2024-02-29). 1 more implementation below.

Abstract

The ever-growing complexity of reinforcement learning (RL) tasks demands a distributed system to efficiently generate and process a massive amount of data. However, existing open-source libraries suffer from various limitations, which impede their practical use in challenging scenarios where large-scale training is necessary. In this paper, we present a novel abstraction on the dataflows of RL training, which unifies diverse RL training applications into a general framework. Following this abstraction, we develop a scalable, efficient, and extensible distributed RL system called ReaLlyScalableRL, which allows efficient and massively parallelized training and easy development of customized algorithms. Our evaluation shows that SRL outperforms existing academic libraries, reaching at most 21x higher training throughput in a distributed setting. On learning performance, beyond performing and scaling well on common RL benchmarks with different RL algorithms, SRL can reproduce the same solution in the challenging hide-and-seek environment as reported by OpenAI with up to 5x speedup in wall-clock time. Notably, SRL is the first in the academic community to perform RL experiments at a large scale with over 15k CPU cores. SRL source code is available at: https://github.com/openpsi-project/srl .

View on arXivView PDF

Code Implementations(2)

openpsi-project/srlOfficial100%

A Really Scalable RL Framework to 10k+ CPUs

384C++, ShellFeb 29, 20242 years ago
distributed-computingdistributed-systemslarge-scalereinforcement-learning
openpsi-projects/srlOfficial100%

SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores

151PythonJul 24, 20232 years ago
deep-learningdistributed-systemsreinforcement-learningrllibscalability

Similar Papers

Highly Parallelized Reinforcement Learning Training with Relaxed Assignment Dependencies

89% match1 repo52025

Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library

85% match1 repo3.4k2025

LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training

85% match2025

Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels

85% match2025

The Art of Scaling Reinforcement Learning Compute for LLMs

84% match2025

MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster

84% match2025

Cite this paper

@article{mei2023srl,
  title  = {SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores},
  author = {Zhiyu Mei and Wei Fu and Jiaxuan Gao and Guangju Wang and Huanchen Zhang and Yi Wu},
  year   = {2023},
  eprint = {2306.16688},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.16688v3},
  journal = {ICLR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.16688.svg)](https://codeofpaper.com/paper/2306.16688)

Discussion