Learning Self-Correction in Vision-Language Models via Rollout Augmentation

Yi Ding, Ziliang Qiu, Bolian Li, Ruqi Zhang
2/9/2026
cs.CVcs.CLcs.LG

No official code found. Best implementation: DripNowhy/Octopus (16 stars, last commit 2026-02-16).

Abstract

Self-correction is essential for solving complex reasoning problems in vision-language models (VLMs). However, existing reinforcement learning (RL) methods struggle to learn it, as effective self-correction behaviors emerge only rarely, making learning signals extremely sparse. To address this challenge, we propose correction-specific rollouts (Octopus), an RL rollout augmentation framework that synthesizes dense self-correction examples by recombining existing rollouts. This augmentation simultaneously improves sample efficiency due to rollout reuse and stabilizes RL optimization through balanced supervision. Furthermore, we introduce a response-masking strategy that decouples self-correction from direct reasoning, avoiding signal conflicts and enabling both behaviors to be learned effectively. Building on this, we introduce Octopus-8B, a reasoning VLM with controllable self-correction capability. Across 7 benchmarks, it achieves SoTA performance among open-source VLMs, outperforming the best RLVR baseline by 1.0 score while requiring only $0.72\times$ training time per step.

View on arXivView PDF

Code Implementations(1)

DripNowhy/OctopusCommunity75%

Official implementation for paper: Learning Self-Correction in Vision–Language Models via Rollout Augmentation

161Jinja, ShellFeb 9, 20268 months ago

Similar Papers

Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks

90% match1 repo22024

Sherlock: Self-Correcting Reasoning in Vision-Language Models

89% match1 repo312025

Training Language Models to Self-Correct via Reinforcement Learning

86% match3 repos1412024

SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning

86% match1 repo22025

VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training

86% match2025

Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme

86% match1 repo1492025

Cite this paper

@article{ding2026learning,
  title  = {Learning Self-Correction in Vision-Language Models via Rollout Augmentation},
  author = {Yi Ding and Ziliang Qiu and Bolian Li and Ruqi Zhang},
  year   = {2026},
  eprint = {2602.08503},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2602.08503v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2602.08503.svg)](https://codeofpaper.com/paper/2602.08503)

Discussion