[2509.22580] The Lie of the Average: How Class Incremental Learning Evaluation Deceives You?

Computer Science > Machine Learning arXiv:2509.22580 (cs) [Submitted on 26 Sep 2025 (v1), last revised 4 Mar 2026 (this version, v2)] Title:The Lie of the Average: How Class Incremental Learning Evaluation Deceives You? Authors:Guannan Lai, Da-Wei Zhou, Xin Yang, Han-Jia Ye View a PDF of the paper titled The Lie of the Average: How Class Incremental Learning Evaluation Deceives You?, by Guannan Lai and 3 other authors View PDF HTML (experimental) Abstract:Class Incremental Learning (CIL) requires models to continuously learn new classes without forgetting previously learned ones, while maintaining stable performance across all possible class sequences. In real-world settings, the order in which classes arrive is diverse and unpredictable, and model performance can vary substantially across different sequences. Yet mainstream evaluation protocols calculate mean and variance from only a small set of randomly sampled sequences. Our theoretical analysis and empirical results demonstrate that this sampling strategy fails to capture the full performance range, resulting in biased mean estimates and a severe underestimation of the true variance in the performance distribution. We therefore contend that a robust CIL evaluation protocol should accurately characterize and estimate the entire performance distribution. To this end, we introduce the concept of extreme sequences and provide theoretical justification for their crucial role in the reliable evaluation of CIL. Moreover, we ...