[2604.00199] QUEST: A robust attention formulation using query-modulated spherical attention

Computer Science > Machine Learning arXiv:2604.00199 (cs) [Submitted on 31 Mar 2026] Title:QUEST: A robust attention formulation using query-modulated spherical attention Authors:Hariprasath Govindarajan, Per Sidén, Jacob Roll, Fredrik Lindsten View a PDF of the paper titled QUEST: A robust attention formulation using query-modulated spherical attention, by Hariprasath Govindarajan and 3 other authors View PDF HTML (experimental) Abstract:The Transformer model architecture has become one of the most widely used in deep learning and the attention mechanism is at its core. The standard attention formulation uses a softmax operation applied to a scaled dot product between query and key vectors. We explore the role played by norms of the queries and keys, which can cause training instabilities when they arbitrarily increase. We demonstrate how this can happen even in simple Transformer models, in the presence of easy-to-learn spurious patterns in the data. We propose a new attention formulation, QUEry-modulated Spherical aTtention (QUEST), that constrains the keys to a hyperspherical latent space, while still allowing individual tokens to flexibly control the sharpness of the attention distribution. QUEST can be easily used as a drop-in replacement for standard attention. We focus on vision applications while also exploring other domains to highlight the method's generality. We show that (1) QUEST trains without instabilities and (2) produces models with improved performance (...