[2510.21356] Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding

Computer Science > Computer Vision and Pattern Recognition arXiv:2510.21356 (cs) [Submitted on 24 Oct 2025 (v1), last revised 24 Mar 2026 (this version, v2)] Title:Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding Authors:Anupam Pani, Yanchao Yang View a PDF of the paper titled Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding, by Anupam Pani and 1 other authors View PDF HTML (experimental) Abstract:Eye gaze offers valuable cues about attention, short-term intent, and future actions, making it a powerful signal for modeling egocentric behavior. In this work, we propose a gaze-regularized framework that enhances VLMs for two key egocentric understanding tasks: fine-grained future event prediction and current activity understanding. Unlike prior approaches that rely solely on visual inputs or use gaze as an auxiliary input signal , our method uses gaze only during training. We introduce a gaze-regularized attention mechanism that aligns model focus with human visual gaze. This design is flexible and modular, allowing it to generalize across multiple VLM architectures that utilize attention. Experimental results show that our approach improves semantic prediction scores by up to 11 for future event prediction and around 7 for current activity understanding, compared to the corresponding baseline models trained without gaze regularization. These results highlight the value of gaze-guided tr...