[2403.04545] Branch Scaling Manifests as Implicit Architectural Regularization for Improving Generalization in Overparameterized ResNets

Computer Science > Machine Learning arXiv:2403.04545 (cs) [Submitted on 7 Mar 2024 (v1), last revised 26 Mar 2026 (this version, v2)] Title:Branch Scaling Manifests as Implicit Architectural Regularization for Improving Generalization in Overparameterized ResNets Authors:Zixiong Yu, Guhan Chen, Jianfa Lai, Bohan Li, Songtao Tian View a PDF of the paper titled Branch Scaling Manifests as Implicit Architectural Regularization for Improving Generalization in Overparameterized ResNets, by Zixiong Yu and 4 other authors View PDF HTML (experimental) Abstract:Scaling factors in residual branches have emerged as a prevalent method for boosting neural network performance, especially in normalization-free architectures. While prior work has primarily examined scaling effects from an optimization perspective, this paper investigates their role in residual architectures through the lens of generalization theory. Specifically, we establish that wide residual networks (ResNets) with constant scaling factors become asymptotically unlearnable as depth increases. In contrast, when the scaling factor exhibits rapid depth-wise decay combined with early stopping, over-parameterized ResNets achieve minimax-optimal generalization rates. To establish this, we demonstrate that the generalization capability of wide ResNets can be approximated by the kernel regression associated with a specific kernel. Our theoretical findings are validated through experiments on synthetic data and real-world class...