[2604.02340] Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models

Computer Science > Machine Learning arXiv:2604.02340 (cs) [Submitted on 4 Feb 2026] Title:Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models Authors:Ivan Sedykh, Nikita Sorokin, Valentin Malykh View a PDF of the paper titled Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models, by Ivan Sedykh and 2 other authors View PDF HTML (experimental) Abstract:Recent advances in masked diffusion language models (MDLMs) narrow the quality gap to autoregressive LMs, but their sampling remains expensive because generation requires many full-sequence denoising passes with a large Transformer and, unlike autoregressive decoding, cannot benefit from KV caching. In this work, we exploit the flexibility of the diffusion framework and study model scheduling, where a smaller MDLM replaces the full model at a subset of denoising steps. On OpenWebText, we show that early and late denoising steps are substantially more robust to such replacement than middle steps, enabling up to a 17% reduction in FLOPs with only modest degradation in generative perplexity. We support these findings with a step-importance analysis based on loss and KL divergence between small and large models across timesteps, as well as an exhaustive search over coarse step segments, both of which identify the middle of the diffusion trajectory as most sensitive. Our results suggest that simple, architecture-agnostic scheduling rules can sign...