[2512.09275] Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression

Statistics > Machine Learning arXiv:2512.09275 (stat) [Submitted on 10 Dec 2025 (v1), last revised 24 Mar 2026 (this version, v2)] Title:Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression Authors:Weiyi He, Yue Xing View a PDF of the paper titled Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression, by Weiyi He and 1 other authors View PDF HTML (experimental) Abstract:Positional encoding (PE) is a core architectural component of Transformers, yet its impact on the Transformer's generalization and robustness remains unclear. In this work, we provide the first generalization analysis for a single-layer Transformer under in-context regression that explicitly accounts for a completely trainable PE module. Our result shows that PE systematically enlarges the generalization gap. Extending to the adversarial setting, we derive the adversarial Rademacher generalization bound. We find that the gap between models with and without PE is magnified under attack, demonstrating that PE amplifies the vulnerability of models. Our bounds are empirically validated by a simulation study. Together, this work establishes a new framework for understanding the clean and adversarial generalization in ICL with PE. Comments: Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG) Cite as: arXiv:2512.09275 [stat.ML]   (or arXiv:2512.09275v2 [stat.ML] for this v...