[2603.24596] X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs

Electrical Engineering and Systems Science > Audio and Speech Processing arXiv:2603.24596 (eess) [Submitted on 6 Mar 2026] Title:X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs Authors:Di Cao, Dongjie Fu, Hai Yu, Siqi Zheng, Xu Tan, Tao Jin View a PDF of the paper titled X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs, by Di Cao and 5 other authors View PDF HTML (experimental) Abstract:While the shift from cascaded dialogue systems to end-to-end (E2E) speech Large Language Models (LLMs) improves latency and paralinguistic modeling, E2E models often exhibit a significant performance degradation compared to their text-based counterparts. The standard Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) training methods fail to close this gap. To address this, we propose X-OPD, a novel Cross-Modal On-Policy Distillation framework designed to systematically align the capabilities of Speech LLMs to their text-based counterparts. X-OPD enables the Speech LLM to explore its own distribution via on-policy rollouts, where a text-based teacher model evaluates these trajectories and provides token-level feedback, effectively distilling teacher's capabilities into student's multi-modal representations. Extensive experiments across multiple benchmarks demonstrate that X-OPD significantly narrows the gap in complex tasks while preserving the model's inherent capabilities. Comments: Subjects: Audio and Speech Pro...