[2603.23805] Deep Neural Regression Collapse

Computer Science > Machine Learning arXiv:2603.23805 (cs) [Submitted on 25 Mar 2026] Title:Deep Neural Regression Collapse Authors:Akshay Rangamani, Altay Unal View a PDF of the paper titled Deep Neural Regression Collapse, by Akshay Rangamani and 1 other authors View PDF HTML (experimental) Abstract:Neural Collapse is a phenomenon that helps identify sparse and low rank structures in deep classifiers. Recent work has extended the definition of neural collapse to regression problems, albeit only measuring the phenomenon at the last layer. In this paper, we establish that Neural Regression Collapse (NRC) also occurs below the last layer across different types of models. We show that in the collapsed layers of neural regression models, features lie in a subspace that corresponds to the target dimension, the feature covariance aligns with the target covariance, the input subspace of the layer weights aligns with the feature subspace, and the linear prediction error of the features is close to the overall prediction error of the model. In addition to establishing Deep NRC, we also show that models that exhibit Deep NRC learn the intrinsic dimension of low rank targets and explore the necessity of weight decay in inducing Deep NRC. This paper provides a more complete picture of the simple structure learned by deep networks in the context of regression. Comments: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE); Machi...