[2603.00133] You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models

Computer Science > Computer Vision and Pattern Recognition arXiv:2603.00133 (cs) [Submitted on 23 Feb 2026] Title:You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models Authors:Kairan Zhao, Eleni Triantafillou, Peter Triantafillou View a PDF of the paper titled You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models, by Kairan Zhao and 2 other authors View PDF HTML (experimental) Abstract:Generative models have been shown to "memorize" certain training data, leading to verbatim or near-verbatim generating images, which may cause privacy concerns or copyright infringement. We introduce Guidance Using Attractive-Repulsive Dynamics (GUARD), a novel framework for memorization mitigation in text-to-image diffusion models. GUARD adjusts the image denoising process to guide the generation away from an original training image and towards one that is distinct from training data while remaining aligned with the prompt, guarding against reproducing training data, without hurting image generation quality. We propose a concrete instantiation of this framework, where the positive target that we steer towards is given by a novel method for (cross) attention attenuation based on (i) a novel statistical mechanism that automatically identifies the prompt positions where cross attention must be attenuated and (ii) attenuating cross-attention in these per-prompt locations. The resulting GUARD offers a...