![[2602.22495] Reinforcement-aware Knowledge Distillation for LLM Reasoning](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.22495] Reinforcement-aware Knowledge Distillation for LLM Reasoning
Summary
The paper presents Reinforcement-aware Knowledge Distillation (RLAD) for enhancing reasoning in large language models (LLMs) by addressing inefficiencies in traditional knowledge distillation methods when combined with reinforcement learning.
Why It Matters
This research is significant as it proposes a novel approach to improve the efficiency of LLMs through RL-aware distillation, which can lead to more effective and resource-efficient AI systems. The findings could have implications for various applications in AI, particularly in enhancing reasoning capabilities in language models.
Key Takeaways
- RLAD improves upon traditional knowledge distillation methods by addressing distribution mismatch and objective interference.
- The core component, Trust Region Ratio Distillation (TRRD), uses a PPO/GRPO-style objective for better alignment between teacher and student models.
- RLAD demonstrates superior performance across diverse logic reasoning and math benchmarks compared to existing methods.
- The approach balances exploration, exploitation, and imitation effectively in the learning process.
- This research contributes to the field of AI by optimizing the distillation process for LLMs, potentially reducing inference costs.
Computer Science > Machine Learning arXiv:2602.22495 (cs) [Submitted on 26 Feb 2026] Title:Reinforcement-aware Knowledge Distillation for LLM Reasoning Authors:Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, Stefano Soatto View a PDF of the paper titled Reinforcement-aware Knowledge Distillation for LLM Reasoning, by Zhaoyang Zhang and 8 other authors View PDF HTML (experimental) Abstract:Reinforcement learning (RL) post-training has recently driven major gains in long chain-of-thought reasoning large language models (LLMs), but the high inference cost of such models motivates distillation into smaller students. Most existing knowledge distillation (KD) methods are designed for supervised fine-tuning (SFT), relying on fixed teacher traces or teacher-student Kullback-Leibler (KL) divergence-based regularization. When combined with RL, these approaches often suffer from distribution mismatch and objective interference: teacher supervision may not align with the student's evolving rollout distribution, and the KL regularizer can compete with reward maximization and require careful loss balancing. To address these issues, we propose RL-aware distillation (RLAD), which performs selective imitation during RL -- guiding the student toward the teacher only when it improves the current policy update. Our core component, Trust Region Ratio Distillation (TRRD), replaces the teacher-student KL regularizer with a PPO/GRPO-style lik...
