![[2602.21919] Learning in the Null Space: Small Singular Values for Continual Learning](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.21919] Learning in the Null Space: Small Singular Values for Continual Learning
Summary
The paper presents NESS, a novel continual learning method that leverages small singular values to maintain orthogonality in weight updates, addressing catastrophic forgetting in machine learning.
Why It Matters
Continual learning is crucial for developing AI systems that can adapt over time without losing previously acquired knowledge. This research provides a new approach that enhances learning efficiency and stability, which is vital for real-world applications in AI.
Key Takeaways
- NESS utilizes small singular values to construct a null space for weight updates.
- The method reduces catastrophic forgetting while enabling adaptation to new tasks.
- Theoretical analysis and experiments show competitive performance across benchmark datasets.
- The approach maintains stability in accuracy, crucial for continual learning applications.
- A single trainable matrix per task simplifies the learning process.
Computer Science > Machine Learning arXiv:2602.21919 (cs) [Submitted on 25 Feb 2026] Title:Learning in the Null Space: Small Singular Values for Continual Learning Authors:Cuong Anh Pham, Praneeth Vepakomma, Samuel Horváth View a PDF of the paper titled Learning in the Null Space: Small Singular Values for Continual Learning, by Cuong Anh Pham and 2 other authors View PDF HTML (experimental) Abstract:Alleviating catastrophic forgetting while enabling further learning is a primary challenge in continual learning (CL). Orthogonal-based training methods have gained attention for their efficiency and strong theoretical properties, and many existing approaches enforce orthogonality through gradient projection. In this paper, we revisit orthogonality and exploit the fact that small singular values correspond to directions that are nearly orthogonal to the input space of previous tasks. Building on this principle, we introduce NESS (Null-space Estimated from Small Singular values), a CL method that applies orthogonality directly in the weight space rather than through gradient manipulation. Specifically, NESS constructs an approximate null space using the smallest singular values of each layer's input representation and parameterizes task-specific updates via a compact low-rank adaptation (LoRA-style) formulation constrained to this subspace. The subspace basis is fixed to preserve the null-space constraint, and only a single trainable matrix is learned for each task. This design...
