[2604.00230] Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold

Computer Science > Machine Learning arXiv:2604.00230 (cs) [Submitted on 31 Mar 2026] Title:Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold Authors:Anamika Paul Rupa View a PDF of the paper titled Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold, by Anamika Paul Rupa View PDF HTML (experimental) Abstract:Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised. We identify a simple and predictive regularity: NC occurs when the mean feature norm reaches a model-dataset-specific critical value, fn*, that is largely invariant to training conditions. This value concentrates tightly within each (model, dataset) pair (CV < 8%); training dynamics primarily affect the rate at which fn approaches fn*, rather than the value itself. In standard training trajectories, the crossing of fn below fn* consistently precedes NC onset, providing a practical predictor with a mean lead time of 62 epochs (MAE 24 epochs). A direct intervention experiment confirms fn* is a stable attractor of the gradient flow -- perturbations to feature scale are self-corrected during training, with convergence to the same value regardless of direction (p>0.2). Completing the (architecture)x(dataset) grid reveals the paper's strongest result: ResNet-20 on MNIST gives fn* = 5.867 -- a +4...