[2309.10370] Geometric structure of shallow neural networks and constructive ${\mathcal L}^2$ cost minimization

Computer Science > Machine Learning arXiv:2309.10370 (cs) [Submitted on 19 Sep 2023 (v1), last revised 26 Feb 2026 (this version, v4)] Title:Geometric structure of shallow neural networks and constructive ${\mathcal L}^2$ cost minimization Authors:Thomas Chen, Patrícia Muñoz Ewald View a PDF of the paper titled Geometric structure of shallow neural networks and constructive ${\mathcal L}^2$ cost minimization, by Thomas Chen and 1 other authors View PDF HTML (experimental) Abstract:In this paper, we approach the problem of cost (loss) minimization in underparametrized shallow ReLU networks through the explicit construction of upper bounds which appeal to the structure of classification data, without use of gradient descent. A key focus is on elucidating the geometric structure of approximate and precise minimizers. We consider an $L^2$ cost function, input space $\mathbb{R}^M$, output space ${\mathbb R}^Q$ with $Q\leq M$, and training input sample size that can be arbitrarily large. We prove an upper bound on the minimum of the cost function of order $O(\delta_P)$ where $\delta_P$ measures the signal-to-noise ratio of training data. In the special case $M=Q$, we explicitly determine an exact degenerate local minimum of the cost function, and show that the sharp value differs from the upper bound obtained for $Q\leq M$ by a relative error $O(\delta_P^2)$. The proof of the upper bound yields a constructively trained network; we show that it metrizes a particular $Q$-dimension...