[2508.13773] PENGUIN: Enhancing Transformer with Periodic-Nested Group Attention for Long-term Time Series Forecasting

Computer Science > Machine Learning arXiv:2508.13773 (cs) [Submitted on 19 Aug 2025 (v1), last revised 29 Mar 2026 (this version, v3)] Title:PENGUIN: Enhancing Transformer with Periodic-Nested Group Attention for Long-term Time Series Forecasting Authors:Tian Sun, Yuqi Chen, Weiwei Sun View a PDF of the paper titled PENGUIN: Enhancing Transformer with Periodic-Nested Group Attention for Long-term Time Series Forecasting, by Tian Sun and 2 other authors View PDF HTML (experimental) Abstract:Despite advances in the Transformer architecture, their effectiveness for long-term time series forecasting (LTSF) remains controversial. In this paper, we investigate the potential of integrating explicit periodicity modeling into the self-attention mechanism to enhance the performance of Transformer-based architectures for LTSF. Specifically, we propose PENGUIN, a simple yet effective periodic-nested group attention mechanism. Our approach introduces a periodic-aware relative attention bias to directly capture periodic structures and a grouped multi-query attention mechanism to handle multiple coexisting periodicities (e.g., daily and weekly cycles) within time series data. Extensive experiments across diverse benchmarks demonstrate that PENGUIN consistently outperforms both MLP-based and Transformer-based models. Code is available at this https URL. Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI) Cite as: arXiv:2508.13773 [cs.LG]   (or arXiv:2508.13773v3 [cs.LG] fo...