[2506.02168] An Approximation Theory Perspective on Machine Learning

Computer Science > Machine Learning arXiv:2506.02168 (cs) [Submitted on 2 Jun 2025 (v1), last revised 4 Mar 2026 (this version, v2)] Title:An Approximation Theory Perspective on Machine Learning Authors:Hrushikesh N. Mhaskar, Efstratios Tsoukanis, Ameya D. Jagtap View a PDF of the paper titled An Approximation Theory Perspective on Machine Learning, by Hrushikesh N. Mhaskar and 2 other authors View PDF HTML (experimental) Abstract:A central problem in machine learning is often formulated as follows: Given a dataset $\{(x_j, y_j)\}_{j=1}^M$, which is a sample drawn from an unknown probability distribution, the goal is to construct a functional model $f$ such that $f(x) \approx y$ for any $(x, y)$ drawn from the same distribution. Neural networks and kernel-based methods are commonly employed for this task due to their capacity for fast and parallel computation. The approximation capabilities, or expressive power, of these methods have been extensively studied over the past 35 years. In this paper, we will present examples of key ideas in this area found in the literature. We will discuss emerging trends in machine learning including the role of shallow/deep networks, approximation on manifolds, physics-informed neural surrogates, neural operators, and transformer architectures. Despite function approximation being a fundamental problem in machine learning, approximation theory does not play a central role in the theoretical foundations of the field. One unfortunate conseque...