![[2602.21501] A Researcher's Guide to Empirical Risk Minimization](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.21501] A Researcher's Guide to Empirical Risk Minimization
Summary
This article provides a comprehensive guide on empirical risk minimization (ERM), detailing high-probability regret bounds and modular presentations for various loss functions and function classes.
Why It Matters
Understanding empirical risk minimization is crucial for researchers and practitioners in machine learning, as it provides foundational insights into model performance and error bounds. This guide offers practical tools and theoretical frameworks that can enhance the reliability of machine learning models, particularly in complex scenarios like causal inference and domain adaptation.
Key Takeaways
- The guide outlines a three-step recipe for deriving regret bounds in ERM.
- It emphasizes the importance of localized Rademacher complexity in defining critical radii.
- The article discusses the implications of nuisance components in various applications, including causal inference.
- It provides concrete upper bounds using local maximal inequalities and metric-entropy integrals.
- The framework allows for regret-transfer bounds linking estimated losses to population regret.
Statistics > Machine Learning arXiv:2602.21501 (stat) [Submitted on 25 Feb 2026] Title:A Researcher's Guide to Empirical Risk Minimization Authors:Lars van der Laan View a PDF of the paper titled A Researcher's Guide to Empirical Risk Minimization, by Lars van der Laan View PDF HTML (experimental) Abstract:This guide develops high-probability regret bounds for empirical risk minimization (ERM). The presentation is modular: we state broadly applicable guarantees under high-level conditions and give tools for verifying them for specific losses and function classes. We emphasize that many ERM rate derivations can be organized around a three-step recipe -- a basic inequality, a uniform local concentration bound, and a fixed-point argument -- which yields regret bounds in terms of a critical radius, defined via localized Rademacher complexity, under a mild Bernstein-type variance--risk condition. To make these bounds concrete, we upper bound the critical radius using local maximal inequalities and metric-entropy integrals, recovering familiar rates for VC-subgraph, Sobolev/Hölder, and bounded-variation classes. We also review ERM with nuisance components -- including weighted ERM and Neyman-orthogonal losses -- as they arise in causal inference, missing data, and domain adaptation. Following the orthogonal learning framework, we highlight that these problems often admit regret-transfer bounds linking regret under an estimated loss to population regret under the target loss. The...
