![[2602.17559] Revisiting Weight Regularization for Low-Rank Continual Learning](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.17559] Revisiting Weight Regularization for Low-Rank Continual Learning
Summary
This paper explores weight regularization techniques in low-rank continual learning, proposing EWC-LoRA to mitigate task interference while maintaining efficiency.
Why It Matters
As continual learning becomes increasingly important in machine learning, understanding how to effectively manage task interference is crucial. This research revisits weight regularization methods, offering a novel approach that enhances the performance of parameter-efficient continual learning, which is vital for deploying large-scale pre-trained models in dynamic environments.
Key Takeaways
- EWC-LoRA combines Elastic Weight Consolidation with low-rank representations to improve continual learning.
- The proposed method maintains constant storage and inference costs regardless of task number.
- Extensive experiments show EWC-LoRA outperforms existing low-rank continual learning methods.
- Weight regularization remains effective even in low-rank parameterizations.
- This research provides insights for broader applications of regularization techniques in continual learning.
Computer Science > Machine Learning arXiv:2602.17559 (cs) [Submitted on 19 Feb 2026] Title:Revisiting Weight Regularization for Low-Rank Continual Learning Authors:Yaoyue Zheng, Yin Zhang, Joost van de Weijer, Gido M van de Ven, Shaoyi Du, Xuetao Zhang, Zhiqiang Tian View a PDF of the paper titled Revisiting Weight Regularization for Low-Rank Continual Learning, by Yaoyue Zheng and 6 other authors View PDF HTML (experimental) Abstract:Continual Learning (CL) with large-scale pre-trained models (PTMs) has recently gained wide attention, shifting the focus from training from scratch to continually adapting PTMs. This has given rise to a promising paradigm: parameter-efficient continual learning (PECL), where task interference is typically mitigated by assigning a task-specific module during training, such as low-rank adapters. However, weight regularization techniques, such as Elastic Weight Consolidation (EWC)-a key strategy in CL-remain underexplored in this new paradigm. In this paper, we revisit weight regularization in low-rank CL as a new perspective for mitigating task interference in PECL. Unlike existing low-rank CL methods, we mitigate task interference by regularizing a shared low-rank update through EWC, thereby keeping the storage requirement and inference costs constant regardless of the number of tasks. Our proposed method EWC-LoRA leverages a low-rank representation to estimate parameter importance over the full-dimensional space. This design offers a practic...
