[2511.10696] $π$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling

Computer Science > Computation and Language arXiv:2511.10696 (cs) [Submitted on 12 Nov 2025 (v1), last revised 28 Mar 2026 (this version, v2)] Title:$π$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling Authors:Dong Liu, Yanxuan Yu View a PDF of the paper titled $\pi$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling, by Dong Liu and 1 other authors View PDF HTML (experimental) Abstract:Transformers have revolutionized natural language processing, but their quadratic complexity with respect to sequence length remains a fundamental bottleneck for long-range modeling. While sparse attention mechanisms like RingAttention reduce computational costs by restricting attention to local neighborhoods, they suffer from limited receptive fields and lack of adaptability. We present \PiAttention, a periodic sparse Transformer that factorizes attention into ring-local neighborhoods, deterministic $\pi$-stride skips, and an adaptive fusion gate. The periodic structure provides predictable coverage of distant tokens, while the sparse footprint keeps the per-layer complexity linear in context length. We prove that \PiAttention achieves $\mathcal{O}(kL + \pi \log L)$ receptive field growth compared to $\mathcal{O}(kL)$ for RingAttention, where $k$ is the local window size, $\pi$ is the skip period, and $L$ is the sequence length. Extensive experiments on language modeling, retrieval, and vision-language tasks demonstrate that \PiAttention ...