[2510.01938] StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold

Computer Science > Machine Learning arXiv:2510.01938 (cs) [Submitted on 2 Oct 2025 (v1), last revised 2 Apr 2026 (this version, v2)] Title:StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold Authors:Zhizhong Li, Sina Sajadmanesh, Jingtao Li, Lingjuan Lyu View a PDF of the paper titled StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold, by Zhizhong Li and 3 other authors View PDF HTML (experimental) Abstract:Low-rank adaptation (LoRA) has been widely adopted as a parameter-efficient technique for fine-tuning large-scale pre-trained models. However, it still lags behind full fine-tuning in performance, partly due to its insufficient exploitation of the geometric structure underlying low-rank manifolds. In this paper, we propose a geometry-aware extension of LoRA that uses a three-factor decomposition $U\!SV^\top$. Analogous to the structure of singular value decomposition (SVD), it separates the adapter's input and output subspaces, $V$ and $U$, from the scaling factor $S$. Our method constrains $U$ and $V$ to lie on the Stiefel manifold, ensuring their orthonormality throughout the training. To optimize on the Stiefel manifold, we employ a flexible and modular geometric optimization design that converts any Euclidean optimizer to a Riemannian one. It enables efficient subspace learning while remaining compatible with existing fine-tuning pipelines. Empirical results across a wide range of downstream tasks, including commonsense reaso...