It Takes Two: Your GRPO Is Secretly DPO

Abstract

Group Relative Policy Optimization (GRPO) has emerged as a prominent reinforcement learning algorithm for post-training Large Language Models. Different from critic-based methods such as PPO, GRPO estimates the advantage function using group-level statistics to reduce the variance of policy gradient estimators. While the prevailing view attributes GRPO's effectiveness to large group sizes for accurate advantage estimation, we propose a different perspective. We demonstrate that the efficacy of GRPO stems from its implicit contrastive objective in the optimization, which helps reduce variance via the control variate method. This perspective establishes a fundamental connection between GRPO and DPO, wherein group size influences only the Monte Carlo estimators of the contrastive objective. To validate this, we investigate the minimal two-rollout case (2-GRPO), a configuration permissible under the contrastive framework but typically considered insufficient for reward normalization. We provide a rigorous theoretical analysis of 2-GRPO and empirically validate its effectiveness: 2-GRPO retains 98.1% of the performance of 16-GRPO, while requiring only 12.5% of the rollouts and 21% of the training time. This study offers a new perspective for future algorithm design in LLM post-training.