[2510.23557] Minimizing Human Intervention in Online Classification

Statistics > Machine Learning arXiv:2510.23557 (stat) [Submitted on 27 Oct 2025 (v1), last revised 1 May 2026 (this version, v2)] Title:Minimizing Human Intervention in Online Classification Authors:William Réveillard, Vasileios Saketos, Alexandre Proutiere, Richard Combes View a PDF of the paper titled Minimizing Human Intervention in Online Classification, by William R\'eveillard and 3 other authors View PDF HTML (experimental) Abstract:Training or fine-tuning large language model (LLM)-based systems often requires costly human feedback, yet there is limited understanding of how to minimize such intervention while maintaining strong error guarantees. We study this problem for LLM-based classification systems in an active learning framework: an agent sequentially labels $d$-dimensional query embeddings drawn i.i.d. from an unknown distribution by either calling a costly expert or guessing with no feedback, with the goal of minimizing regret relative to an oracle with free expert access. When the horizon $T$ is at least exponential in the embedding dimension $d$, the geometry of the class regions can be learned. In this regime, we propose the Conservative Hull-based Classifier (CHC), which maintains convex hulls of expert-labeled queries and calls the expert when a query lands outside all known hulls. CHC attains $\mathcal{O}(\log^d T)$ regret in $T$ and is minimax optimal for $d=1$. Otherwise, the geometry cannot be reliably learned in general. We show that for queries dr...