![[2505.11235] Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2505.11235] Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation
Summary
The paper presents Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation (PSOFT), a method that enhances parameter-efficient fine-tuning by preserving semantic representations while improving expressiveness and efficiency across various tasks.
Why It Matters
As large models become increasingly prevalent, efficient fine-tuning methods are crucial for adapting these models to specific tasks without excessive computational costs. PSOFT addresses the limitations of existing methods by maintaining semantic integrity while optimizing performance, making it a significant contribution to the field of machine learning.
Key Takeaways
- PSOFT confines orthogonal transformations to the principal subspace of pre-trained weights.
- The method enhances adaptability by gradually relaxing orthogonality during training.
- Extensive experiments demonstrate PSOFT's effectiveness across 35 NLP and CV tasks.
- PSOFT achieves a balance between semantic preservation and computational efficiency.
- The code for PSOFT is publicly available, promoting further research and application.
Computer Science > Machine Learning arXiv:2505.11235 (cs) [Submitted on 16 May 2025 (v1), last revised 19 Feb 2026 (this version, v3)] Title:Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation Authors:Fei Wu, Jia Hu, Geyong Min, Shiqiang Wang View a PDF of the paper titled Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation, by Fei Wu and 3 other authors View PDF HTML (experimental) Abstract:Driven by the rapid growth of model parameters, parameter-efficient fine-tuning (PEFT) has become essential for adapting large models to diverse downstream tasks under constrained computational resources. Within this paradigm, orthogonal fine-tuning and its variants preserve semantic representations of pre-trained models, but struggle to achieve both expressiveness and efficiency in terms of parameter counts, memory, and computation. To overcome this limitation, we propose efficient Orthogonal Fine-Tuning with Principal Subspace adaptation (PSOFT), which confines orthogonal transformations to the principal subspace of pre-trained weights. Specifically, PSOFT constructs this subspace via matrix decomposition to enable compatible transformations with higher effective rank, establishes a theoretical condition that strictly maintains the geometry of this subspace for essential semantic preservation, and introduces efficient tunable vectors that gradually relax orthogonality during training to enhance adaptability. Extensive experiments on 35 NLP and CV tasks...