![[2602.19619] Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.19619] Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models
Summary
This article evaluates the accuracy of discrete diffusion language models (dLLMs) through a sampler-centric framework, revealing significant errors in few-step samplers compared to autoregressive models.
Why It Matters
Understanding the limitations of dLLMs is crucial for researchers and practitioners in machine learning, particularly in generative AI, as it informs the development of more accurate sampling methods and metrics. This evaluation highlights the need for improved methodologies in assessing model performance.
Key Takeaways
- Discrete diffusion language models may not be distributionally correct even with optimal denoisers.
- Existing evaluation metrics can misrepresent the performance of samplers.
- Improvements in certain metrics do not guarantee correct sampling outcomes.
- The study introduces a novel oracle framework for isolating sampler-induced errors.
- Few-step samplers show significant transition-level mismatches that require more steps to resolve.
Computer Science > Machine Learning arXiv:2602.19619 (cs) [Submitted on 23 Feb 2026] Title:Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models Authors:Luhan Tang, Longxuan Yu, Shaorong Zhang, Greg Ver Steeg View a PDF of the paper titled Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models, by Luhan Tang and 3 other authors View PDF HTML (experimental) Abstract:Discrete diffusion language models (dLLMs) provide a fast and flexible alternative to autoregressive models (ARMs) via iterative denoising with parallel updates. However, their evaluation is challenging: existing metrics conflate denoiser approximation error with sampler-induced error from the sampling dynamics, a problem that does not arise for ARMs whose autoregressive sampling exactly reflects the learned probability model. We introduce a sampler-centric oracle framework that replaces learned denoisers with an exact Hidden Markov Model posterior derived from a ground-truth Markov chain, isolating sampler-induced error in a controlled setting. We show that few-step discrete diffusion samplers are not distributionally correct even under an oracle denoiser, with transition-level mismatch that vanishes only as the number of steps approaches the sequence length. Moreover, improvements in negative log-likelihood, generative perplexity, or MAUVE do not imply correct sampling. Code is available at this htt...
