Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

Kaixiang Zhao, Tianrun Yu, Shangzhe Li, Yuxiao Yang, Amanda Hughes +2 more
10/14/2026
cs.LGcs.AI

No confident code implementation found for this paper.

Abstract

On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets

89% match2026

ReTeach: Building a Self-Teacher through Multi-Round Reflection and Retry

89% match2026

Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation

88% match2026

Interactive-Policy Distillation with Bidirectional Propose-and-Verify

88% match2026

From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation

87% match1 repo2026

ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation

87% match1 repo22026

Cite this paper

@article{zhao2026recovering,
  title  = {Recovering Reliable Supervision via Critique-Guided On-Policy Distillation},
  author = {Kaixiang Zhao and Tianrun Yu and Shangzhe Li and Yuxiao Yang and Amanda Hughes and Weitong Zhang and Taylor W. Killian},
  year   = {2026},
  eprint = {2610.20572},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2610.20572v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2610.20572.svg)](https://codeofpaper.com/paper/2610.20572)

Discussion