[2603.04895] How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?

Statistics > Machine Learning arXiv:2603.04895 (stat) [Submitted on 5 Mar 2026] Title:How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression? Authors:Kuo-Wei Lai, Guanghui Wang, Molei Tao, Vidya Muthukumar View a PDF of the paper titled How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?, by Kuo-Wei Lai and 3 other authors View PDF HTML (experimental) Abstract:Overparameterized ML models, including neural networks, typically induce underdetermined training objectives with multiple global minima. The implicit bias refers to the limiting global minimum that is attained by a common optimization algorithm, such as gradient descent (GD). In this paper, we characterize the implicit bias of GD for training a shallow ReLU model with the squared loss on high-dimensional random features. Prior work showed that the implicit bias does not exist in the worst-case (Vardi and Shamir, 2021), or corresponds exactly to the minimum-l2-norm solution among all global minima under exactly orthogonal data (Boursier et al., 2022). Our work interpolates between these two extremes and shows that, for sufficiently high-dimensional random data, the implicit bias approximates the minimum-l2-norm solution with high probability with a gap on the order $\Theta(\sqrt{n/d})$, where n is the number of training examples and d is the feature dimension. Our results are obtained ...