[2603.04064] Tuning Just Enough: Lightweight Backdoor Attacks on Multi-Encoder Diffusion Models

Computer Science > Machine Learning arXiv:2603.04064 (cs) [Submitted on 4 Mar 2026] Title:Tuning Just Enough: Lightweight Backdoor Attacks on Multi-Encoder Diffusion Models Authors:Ziyuan Chen, Yujin Jeong, Tobias Braun, Anna Rohrbach View a PDF of the paper titled Tuning Just Enough: Lightweight Backdoor Attacks on Multi-Encoder Diffusion Models, by Ziyuan Chen and 3 other authors View PDF HTML (experimental) Abstract:As text-to-image diffusion models become increasingly deployed in real-world applications, concerns about backdoor attacks have gained significant attention. Prior work on text-based backdoor attacks has largely focused on diffusion models conditioned on a single lightweight text encoder. However, more recent diffusion models that incorporate multiple large-scale text encoders remain underexplored in this context. Given the substantially increased number of trainable parameters introduced by multiple text encoders, an important question is whether backdoor attacks can remain both efficient and effective in such settings. In this work, we study Stable Diffusion 3, which uses three distinct text encoders and has not yet been systematically analyzed for text-encoder-based backdoor vulnerabilities. To understand the role of text encoders in backdoor attacks, we define four categories of attack targets and identify the minimal sets of encoders required to achieve effective performance for each attack objective. Based on this, we further propose Multi-Encoder Ligh...