[2507.10610] LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agents

Computer Science > Cryptography and Security arXiv:2507.10610 (cs) [Submitted on 13 Jul 2025 (v1), last revised 31 Mar 2026 (this version, v2)] Title:LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agents Authors:Zihe Yan, Zhuosheng Zhang, Jiaping Gui, Gongshen Liu View a PDF of the paper titled LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agents, by Zihe Yan and 3 other authors View PDF HTML (experimental) Abstract:Graphical user interface (GUI) agents built on multimodal large language models (MLLMs) have recently demonstrated strong decision-making abilities in screen-based interaction tasks. However, they remain highly vulnerable to pop-up-based environmental injection attacks, where malicious visual elements divert model attention and lead to unsafe or incorrect actions. Existing defense methods either require costly retraining or perform poorly under inductive interference. In this work, we systematically study how such attacks alter the attention behavior of GUI agents and uncover a layer-wise attention divergence pattern between correct and incorrect outputs. Based on this insight, we propose \textbf{LaSM}, a \textit{Layer-wise Scaling Mechanism} that selectively amplifies attention and MLP modules in critical layers. LaSM improves the alignment between model saliency and task-relevant regions without additional training. Extensive experiments across multiple datasets demonstrate that our method significantly improves ...