[2511.20592] Latent Diffusion Inversion Requires Understanding the Latent Space

Computer Science > Machine Learning arXiv:2511.20592 (cs) [Submitted on 25 Nov 2025 (v1), last revised 24 Mar 2026 (this version, v2)] Title:Latent Diffusion Inversion Requires Understanding the Latent Space Authors:Mingxing Rao, Bowen Qu, Daniel Moyer View a PDF of the paper titled Latent Diffusion Inversion Requires Understanding the Latent Space, by Mingxing Rao and 2 other authors View PDF HTML (experimental) Abstract:The recovery of training data from generative models ("model inversion") has been extensively studied for diffusion models in the data domain as a memorization/overfitting phenomenon. Latent diffusion models (LDMs), which operate on the latent codes from encoder/decoder pairs, have been robust to prior inversion methods. In this work we describe two key findings: (1) the diffusion model exhibits non-uniform memorization across latent codes, tending to overfit samples located in high-distortion regions of the decoder pullback metric; (2) even within a single latent code, memorization contributions are unequal across representation dimensions. Our proposed method to ranks latent dimensions by their contribution to the decoder pullback metric, which in turn identifies dimensions that contribute to memorization. For score-based membership inference, a sub-task of model inversion, we find that removing less-memorizing dimensions improves performance on all tested methods and datasets, with average AUROC gains of 1-4% and substantial increases in TPR@1%FPR (1-3...