Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs

Mengqi Liao, Xiangyu Xi, Ruinian Chen, Jia Leng, Yangen Hu +3 more
5/24/2025
cs.LGcs.CL

Official code: LiaoMengqi/E3-RL4LLMs (17 stars, last commit 2025-11-07).

Abstract

Reasoning large language models (LLMs) excel in complex tasks, which has drawn significant attention to reinforcement learning (RL) for LLMs. However, existing approaches allocate an equal number of rollouts to all questions during the RL process, which is inefficient. This inefficiency stems from the fact that training on simple questions yields limited gains, whereas more rollouts are needed for challenging questions to sample correct answers. Furthermore, while RL improves response precision, it limits the model's exploration ability, potentially resulting in a performance cap below that of the base model prior to RL. To address these issues, we propose a mechanism for dynamically allocating rollout budgets based on the difficulty of the problems, enabling more efficient RL training. Additionally, we introduce an adaptive dynamic temperature adjustment strategy to maintain the entropy at a stable level, thereby encouraging sufficient exploration. This enables LLMs to improve response precision while preserving their exploratory ability to uncover potential correct pathways. The code and data is available on: https://github.com/LiaoMengqi/E3-RL4LLMs

View on arXivView PDF

Code Implementations(1)

LiaoMengqi/E3-RL4LLMsOfficial100%

[ EMNLP 2025 Main ] Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs

171Roff, ShellMay 18, 202511 months ago

Similar Papers

Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay

90% match1 repo462025

Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay

90% match1 repo462025

Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations

89% match2025

Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration

89% match2025

How Can LLM Guide RL? A Value-Based Approach

88% match1 repo212024

From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR

88% match2025

Cite this paper

@article{liao2025enhancing,
  title  = {Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs},
  author = {Mengqi Liao and Xiangyu Xi and Ruinian Chen and Jia Leng and Yangen Hu and Ke Zeng and Shuai Liu and Huaiyu Wan},
  year   = {2025},
  eprint = {2505.18573},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.18573v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.18573.svg)](https://codeofpaper.com/paper/2505.18573)

Discussion