[2505.11006] Is Supervised Learning Really That Different from Unsupervised?

Statistics > Machine Learning arXiv:2505.11006 (stat) [Submitted on 16 May 2025 (v1), last revised 27 Mar 2026 (this version, v5)] Title:Is Supervised Learning Really That Different from Unsupervised? Authors:Oskar Allerbo, Thomas B. Schön View a PDF of the paper titled Is Supervised Learning Really That Different from Unsupervised?, by Oskar Allerbo and Thomas B. Sch\"on View PDF HTML (experimental) Abstract:We demonstrate how supervised learning can be decomposed into a two-stage procedure, where (1) all model parameters are selected in an unsupervised manner, and (2) the outputs y are added to the model, without changing the parameter values. This is achieved by a new model selection criterion that - in contrast to cross-validation - can be used also without access to y. For linear ridge regression, we bound the asymptotic out-of-sample risk of our method in terms of the optimal asymptotic risk. We also demonstrate that versions of linear and kernel ridge regression, smoothing splines, k-nearest neighbors, random forests, and neural networks, trained without access to y, perform similarly to their standard y-based counterparts. Hence, our results suggest that the difference between supervised and unsupervised learning is less fundamental than it may appear. Comments: Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG) Cite as: arXiv:2505.11006 [stat.ML]   (or arXiv:2505.11006v5 [stat.ML] for this version)   https://doi.org/10.48550/arXiv.2505.11006 Focus to l...