GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models

Jixiao Zhang, Chunsheng Zuo
4/13/2025
cs.CL

Official code: aeroplanepaper/GRPO-LEAD (38 stars, Apache-2.0, last commit 2025-11-18).

Abstract

Group Relative Policy Optimization (GRPO), which is widely adopted by R1-like reasoning models, has advanced mathematical reasoning. Nevertheless, GRPO faces challenges in reward sparsity, verbosity, and inadequate focus on problem difficulty. We propose GRPO-LEAD, enhancing GRPO with: (1) length-regularized rewards to encourage conciseness while maintaining accuracy; (2) explicit penalties for incorrect solutions to improve model precision; and (3) difficulty-aware advantage reweighting for robust generalization on challenging problems. Comprehensive evaluations demonstrate that GRPO-LEAD significantly improves reasoning accuracy, conciseness, and efficiency. Our approach achieves state-of-the-art performance for 14B-scale models, underscoring the synergy of our methods with appropriate model scale and high-quality data. Our source code, generated dataset, and models are available at https://github.com/aeroplanepaper/GRPO-LEAD.

View on arXivView PDF

Code Implementations(1)

aeroplanepaper/GRPO-LEADOfficial100%
Apache-2.0
381Shell, PythonApr 12, 202511 months agoApache-2.0

Similar Papers

DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage

90% match1 repo42026

Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation

89% match1 repo1292026

Prompt Augmentation Scales up GRPO Training on Mathematical Reasoning

89% match1 repo132026

Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning

89% match1 repo212025

GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning

89% match2025

Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO

88% match2025

Cite this paper

@article{zhang2025grpolead,
  title  = {GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models},
  author = {Jixiao Zhang and Chunsheng Zuo},
  year   = {2025},
  eprint = {2504.09696},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2504.09696v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2504.09696.svg)](https://codeofpaper.com/paper/2504.09696)

Discussion