[2406.03736] Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data

Computer Science > Machine Learning arXiv:2406.03736 (cs) [Submitted on 6 Jun 2024 (v1), last revised 23 Mar 2026 (this version, v4)] Title:Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data Authors:Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, Chongxuan Li View a PDF of the paper titled Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data, by Jingyang Ou and 6 other authors View PDF HTML (experimental) Abstract:Discrete diffusion models with absorbing processes have shown promise in language modeling. The key quantities to be estimated are the ratios between the marginal probabilities of two transitive states at all timesteps, called the concrete score. In this paper, we reveal that the concrete score in absorbing diffusion can be expressed as conditional probabilities of clean data, multiplied by a time-dependent scalar in an analytic form. Motivated by this finding, we propose reparameterized absorbing discrete diffusion (RADD), a dedicated diffusion model without time-condition that characterizes the time-independent conditional probabilities. Besides its simplicity, RADD can reduce the number of function evaluations (NFEs) by caching the output of the time-independent network when the noisy sample remains unchanged in a sampling interval, which enables sampling acceleration. Built upon the new perspective of conditional distributions, we further unify absorbi...