[2603.20898] Natural Gradient Descent for Online Continual Learning

Computer Science > Machine Learning arXiv:2603.20898 (cs) [Submitted on 21 Mar 2026] Title:Natural Gradient Descent for Online Continual Learning Authors:Joe Khawand, David Colliaux View a PDF of the paper titled Natural Gradient Descent for Online Continual Learning, by Joe Khawand and 1 other authors View PDF HTML (experimental) Abstract:Online Continual Learning (OCL) for image classification represents a challenging subset of Continual Learning, focusing on classifying images from a stream without assuming data independence and identical distribution (i.i.d). The primary challenge in this context is to prevent catastrophic forgetting, where the model's performance on previous tasks deteriorates as it learns new ones. Although various strategies have been proposed to address this issue, achieving rapid convergence remains a significant challenge in the online setting. In this work, we introduce a novel approach to training OCL models that utilizes the Natural Gradient Descent optimizer, incorporating an approximation of the Fisher Information Matrix (FIM) through Kronecker Factored Approximate Curvature (KFAC). This method demonstrates substantial improvements in performance across all OCL methods, particularly when combined with existing OCL tricks, on datasets such as Split CIFAR-100, CORE50, and Split miniImageNet. Comments: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) Cite as: arXiv:2603.20898 [...