[2603.03084] On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functions

Computer Science > Machine Learning arXiv:2603.03084 (cs) [Submitted on 3 Mar 2026] Title:On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functions Authors:Linyan Gu, Lihua Yang, Feng Zhou View a PDF of the paper titled On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functions, by Linyan Gu and 2 other authors View PDF HTML (experimental) Abstract:Transformer networks have achieved remarkable empirical success across a wide range of applications, yet their theoretical expressive power remains insufficiently understood. In this paper, we study the expressive capabilities of Transformer architectures. We first establish an explicit approximation of maxout networks by Transformer networks while preserving comparable model complexity. As a consequence, Transformers inherit the universal approximation capability of ReLU networks under similar complexity constraints. Building on this connection, we develop a framework to analyze the approximation of continuous piecewise linear functions by Transformers and quantitatively characterize their expressivity via the number of linear regions, which grows exponentially with depth. Our analysis establishes a theoretical bridge between approximation theory for standard feedforward neural networks and Transformer architectures. It also yields structural insights into Transformers: self-attention layers implement max-type operations, while feedforward la...