![[2602.16456] Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.16456] Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC
Summary
This paper presents a novel approach called LoRSum for optimizing Low-Rank Adaptation (LoRA) in machine learning, enhancing efficiency in training large models without full-matrix SVD projections.
Why It Matters
As machine learning models grow in complexity, efficient training methods become crucial. This research addresses the limitations of existing LoRA techniques, offering a more memory-efficient alternative that can improve performance while reducing computational overhead.
Key Takeaways
- LoRSum optimizes LoRA by treating it as a proximal sub-problem.
- The method achieves efficiency without relying on full-matrix SVD projections.
- Experiments show that LoRSum can match or exceed LoRA baselines with modest compute costs.
- The approach incorporates structured metrics like K-FAC for improved performance.
- LoRSum can also update low-rank momentum, enhancing its versatility.
Computer Science > Machine Learning arXiv:2602.16456 (cs) [Submitted on 18 Feb 2026] Title:Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC Authors:Abdulla Jasem Almansoori, Maria Ivanova, Andrey Veprikov, Aleksandr Beznosikov, Samuel Horváth, Martin Takáč View a PDF of the paper titled Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC, by Abdulla Jasem Almansoori and Maria Ivanova and Andrey Veprikov and Aleksandr Beznosikov and Samuel Horv\'ath and Martin Tak\'a\v{c} View PDF HTML (experimental) Abstract:Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. In this work, we address the gap between training with full steps with low-rank projections (SVDLoRA) and LoRA fine-tuning. We propose LoRSum, a memory-efficient subroutine that closes this gap for gradient descent by casting LoRA optimization as a proximal sub-problem and solving it efficiently with alternating least squares updates, which we prove to be an implicit block power method. We recover several recently proposed preconditioning methods for LoRA as special cases, and show that LoRSum can also be used for updating a low-rank momentum. In order to address full steps with preconditioned gradient descent, we propose a scaled variant of LoRSum that uses structured metrics such as K-FAC and Shampoo, and we show that storing the dia...
