[2603.25562] Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes

Computer Science > Machine Learning arXiv:2603.25562 (cs) [Submitted on 26 Mar 2026] Title:Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes Authors:Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Yuanheng Zhu, Dongbin Zhao View a PDF of the paper titled Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes, by Yuqian Fu and 4 other authors View PDF HTML (experimental) Abstract:On-policy distillation (OPD) is appealing for large language model (LLM) post-training because it evaluates teacher feedback on student-generated rollouts rather than fixed teacher traces. In long-horizon settings, however, the common sampled-token variant is fragile: it reduces distribution matching to a one-token signal and becomes increasingly unreliable as rollouts drift away from prefixes the teacher commonly visits. We revisit OPD from the estimator and implementation sides. Theoretically, token-level OPD is biased relative to sequence-level reverse-KL, but it has a much tighter worst-case variance bound; our toy study shows the same tradeoff empirically, with stronger future-reward coupling producing higher gradient variance and less stable learning. Empirically, we identify three failure modes of sampled-token OPD: an imbalanced one-token signal, unreliable teacher guidance on student-generated prefixes, and distortions caused by tokenizer or special-token mismatch. We address these issues with teacher top-K local support matching, implemented as trunc...