[2603.05498] The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks

Computer Science > Artificial Intelligence arXiv:2603.05498 (cs) [Submitted on 5 Mar 2026] Title:The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks Authors:Shangwen Sun, Alfredo Canziani, Yann LeCun, Jiachen Zhu View a PDF of the paper titled The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks, by Shangwen Sun and 3 other authors View PDF HTML (experimental) Abstract:We study two recurring phenomena in Transformer language models: massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks, in which certain tokens attract disproportionate attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same tokens, but their functional roles and causal relationship remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally: they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters of the model. Attention sinks operate locally: they modulate attention outputs across heads and bias individual heads toward short-range dependencies. We identify the pre-norm configuration as the key choice that enables the co-occurrence, and show that...