[2510.16051] GUIrilla: A Scalable Framework for Automated Desktop UI Exploration

Computer Science > Machine Learning arXiv:2510.16051 (cs) [Submitted on 16 Oct 2025 (v1), last revised 24 Mar 2026 (this version, v2)] Title:GUIrilla: A Scalable Framework for Automated Desktop UI Exploration Authors:Sofiya Garkot, Maksym Shamrai, Ivan Synytsia, Mariya Hirna View a PDF of the paper titled GUIrilla: A Scalable Framework for Automated Desktop UI Exploration, by Sofiya Garkot and 3 other authors View PDF HTML (experimental) Abstract:The performance and generalization of foundation models for interactive systems critically depend on the availability of large-scale, realistic training data. While recent advances in large language models (LLMs) have improved GUI understanding, progress in desktop automation remains constrained by the scarcity of high-quality, publicly available desktop interaction data, particularly for macOS. We introduce GUIRILLA, a scalable data crawling framework for automated exploration of desktop GUIs. GUIRILLA is not an autonomous agent; instead, it systematically collects realistic interaction traces and accessibility metadata intended to support the training, evaluation, and stabilization of downstream foundation models and GUI agents. The framework targets macOS, a largely underrepresented platform in existing resources, and organizes explored interfaces into hierarchical MacApp Trees derived from accessibility states and user actions. As part of this work, we release these MacApp Trees as a reusable structural representation of macOS...