[2511.15927] DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone

Computer Science > Machine Learning arXiv:2511.15927 (cs) [Submitted on 19 Nov 2025 (v1), last revised 27 Feb 2026 (this version, v3)] Title:DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone Authors:Vaibhav Singh, Oleksiy Ostapenko, Pierre-André Noël, Eugene Belilovsky, Torsten Scholak View a PDF of the paper titled DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone, by Vaibhav Singh and 4 other authors View PDF HTML (experimental) Abstract:Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2x and 4.3x higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a p...