First Return, Entropy-Eliciting Explore

Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu +8 more
7/9/2025
cs.AI

No confident code implementation found for this paper.

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) improves the reasoning abilities of Large Language Models (LLMs) but it struggles with unstable exploration. We propose FR3E (First Return, Entropy-Eliciting Explore), a structured exploration framework that identifies high-uncertainty decision points in reasoning trajectories and performs targeted rollouts to construct semantically grounded intermediate feedback. Our method provides targeted guidance without relying on dense supervision. Empirical results on mathematical reasoning benchmarks(AIME24) show that FR3E promotes more stable training, produces longer and more coherent responses, and increases the proportion of fully correct trajectories. These results highlight the framework's effectiveness in improving LLM reasoning through more robust and structured exploration.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning

89% match2025

G$^2$RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance

88% match1 repo162025

Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning

86% match1 repo482025

From Reasoning to Super-Intelligence: A Search-Theoretic Perspective

86% match2025

Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning

86% match2026

ProCeedRL: Process Critic with Exploratory Demonstration Reinforcement Learning for LLM Agentic Reasoning

86% match2026

Cite this paper

@article{zheng2025first,
  title  = {First Return, Entropy-Eliciting Explore},
  author = {Tianyu Zheng and Tianshun Xing and Qingshui Gu and Taoran Liang and Xingwei Qu and Xin Zhou and Yizhi Li and Zhoufutu Wen and Chenghua Lin and Wenhao Huang and Qian Liu and Ge Zhang and Zejun Ma},
  year   = {2025},
  eprint = {2507.07017},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2507.07017v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2507.07017.svg)](https://codeofpaper.com/paper/2507.07017)

Discussion