[2604.05253] Spike Hijacking in Late-Interaction Retrieval

Computer Science > Information Retrieval arXiv:2604.05253 (cs) [Submitted on 6 Apr 2026] Title:Spike Hijacking in Late-Interaction Retrieval Authors:Karthik Suresh, Tushar Vatsa, Tracy King, Asim Kadav, Michael Friedrich View a PDF of the paper titled Spike Hijacking in Late-Interaction Retrieval, by Karthik Suresh and 4 other authors View PDF HTML (experimental) Abstract:Late-interaction retrieval models rely on hard maximum similarity (MaxSim) to aggregate token-level similarities. Although effective, this winner-take-all pooling rule may structurally bias training dynamics. We provide a mechanistic study of gradient routing and robustness in MaxSim-based retrieval. In a controlled synthetic environment with in-batch contrastive training, we demonstrate that MaxSim induces significantly higher patch-level gradient concentration than smoother alternatives such as Top-k pooling and softmax aggregation. While sparse routing can improve early discrimination, it also increases sensitivity to document length: as the number of document patches grows, MaxSim degrades more sharply than mild smoothing variants. We corroborate these findings on a real-world multi-vector retrieval benchmark, where controlled document-length sweeps reveal similar brittleness under hard max pooling. Together, our results isolate pooling-induced gradient concentration as a structural property of late-interaction retrieval and highlight a sparsity-robustness tradeoff. These findings motivate principled ...