SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning

Ruiqi Zhang, Daman Arora, Song Mei, Andrea Zanette
6/10/2025
cs.LG

No official code found. Best implementation: Zanette-Labs/speed-rl (18 stars, last commit 2026-02-02).

Abstract

Training large language models with reinforcement learning (RL) against verifiable rewards significantly enhances their reasoning abilities, yet remains computationally expensive due to inefficient uniform prompt sampling. We introduce Selective Prompting with Efficient Estimation of Difficulty (SPEED), an adaptive online RL curriculum that selectively chooses training examples of intermediate difficulty to maximize learning efficiency. Theoretically, we establish that intermediate-difficulty prompts improve the gradient estimator's signal-to-noise ratio, accelerating convergence. Empirically, our efficient implementation leads to 2x to 6x faster training without degrading accuracy, requires no manual tuning, and integrates seamlessly into standard RL algorithms.

View on arXivView PDF

Code Implementations(1)

Zanette-Labs/speed-rlCommunity100%
183CSS, HTMLJun 9, 20258 months ago

Similar Papers

Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning

88% match2025

RAPID: An Efficient Reinforcement Learning Algorithm for Small Language Models

88% match2025

Reinforcement Learning is all You Need

87% match2025

Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models

87% match1 repo2025

Prompt Curriculum Learning for Efficient LLM Post-Training

87% match2025

Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning

87% match1 repo242025

Cite this paper

@article{zhang2025speedrl,
  title  = {SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning},
  author = {Ruiqi Zhang and Daman Arora and Song Mei and Andrea Zanette},
  year   = {2025},
  eprint = {2506.09016},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.09016v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.09016.svg)](https://codeofpaper.com/paper/2506.09016)

Discussion