[2603.19290] Neural Dynamics Self-Attention for Spiking Transformers

Computer Science > Neural and Evolutionary Computing arXiv:2603.19290 (cs) [Submitted on 9 Mar 2026] Title:Neural Dynamics Self-Attention for Spiking Transformers Authors:Dehao Zhang, Fukai Guo, Shuai Wang, Jingya Wang, Jieyuan Zhang, Yimeng Shan, Malu Zhang, Yang Yang, Haizhou Li View a PDF of the paper titled Neural Dynamics Self-Attention for Spiking Transformers, by Dehao Zhang and Fukai Guo and Shuai Wang and Jingya Wang and Jieyuan Zhang and Yimeng Shan and Malu Zhang and Yang Yang and Haizhou Li View PDF HTML (experimental) Abstract:Integrating Spiking Neural Networks (SNNs) with Transformer architectures offers a promising pathway to balance energy efficiency and performance, particularly for edge vision applications. However, existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference. Through theoretical analysis, we attribute both limitations to the Spiking Self-Attention (SSA) mechanism: the lack of locality bias and the need to store large attention matrices. Inspired by the localized receptive fields (LRF) and membrane-potential dynamics of biological visual neurons, we propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements. Specifically, we introduce a LRF method into SSA to assign higher weights to neighboring regions, strengthening local...