[2601.16884] Multigrade Neural Network Approximation

Computer Science > Machine Learning arXiv:2601.16884 (cs) [Submitted on 23 Jan 2026 (v1), last revised 2 Apr 2026 (this version, v2)] Title:Multigrade Neural Network Approximation Authors:Shijun Zhang, Zuowei Shen, Yuesheng Xu View a PDF of the paper titled Multigrade Neural Network Approximation, by Shijun Zhang and 2 other authors View PDF HTML (experimental) Abstract:We study multigrade deep learning (MGDL) as a principled framework for structured error refinement in deep neural networks. While the approximation power of neural networks is now relatively well understood, training very deep architectures remains challenging due to highly non-convex and often ill-conditioned optimization landscapes. In contrast, for relatively shallow networks, most notably one-hidden-layer $\texttt{ReLU}$ models, training admits convex reformulations with global guarantees, motivating learning paradigms that improve stability while scaling to depth. MGDL builds upon this insight by training deep networks grade by grade: previously learned grades are frozen, and each new residual block is trained solely to reduce the remaining approximation error, yielding an interpretable and stable hierarchical refinement process. We develop an operator-theoretic foundation for MGDL and prove that, for any continuous target function, there exists a fixed-width multigrade $\texttt{ReLU}$ scheme whose residuals decrease strictly across grades and converge uniformly to zero. To the best of our knowledge, t...