[2502.21278] Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion

Computer Science > Machine Learning arXiv:2502.21278 (cs) [Submitted on 28 Feb 2025 (v1), last revised 1 Mar 2026 (this version, v2)] Title:Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion Authors:Kulin Shah, Alkis Kalavasis, Adam R. Klivans, Giannis Daras View a PDF of the paper titled Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusion, by Kulin Shah and 3 other authors View PDF HTML (experimental) Abstract:There is strong empirical evidence that the state-of-the-art diffusion modeling paradigm leads to models that memorize the training set, especially when the training set is small. Prior methods to mitigate the memorization problem often lead to a decrease in image quality. Is it possible to obtain strong and creative generative models, i.e., models that achieve high generation quality and low memorization? Despite the current pessimistic landscape of results, we make significant progress in pushing the trade-off between fidelity and memorization. We first provide theoretical evidence that memorization in diffusion models is only necessary for denoising problems at low noise scales (usually used in generating high-frequency details). Using this theoretical insight, we propose a simple, principled method to train the diffusion models using noisy data at large noise scales. We show that our method significantly reduces memorization without decreasing the image quality, for both text-conditional an...