![[2602.22911] NoRA: Breaking the Linear Ceiling of Low-Rank Adaptation via Manifold Expansion](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.22911] NoRA: Breaking the Linear Ceiling of Low-Rank Adaptation via Manifold Expansion
Summary
The paper introduces NoRA, a novel approach to Low-Rank Adaptation (LoRA) that overcomes the limitations of linear methods by utilizing manifold expansion techniques, demonstrating superior performance in complex reasoning tasks.
Why It Matters
NoRA addresses critical limitations in existing Low-Rank Adaptation methods, particularly in handling complex reasoning tasks. By enhancing spectral efficiency and preventing rank collapse, this research could significantly improve the performance of machine learning models in various applications, making it a valuable contribution to the field.
Key Takeaways
- NoRA introduces a non-linear approach to Low-Rank Adaptation, enhancing performance in complex reasoning tasks.
- The method outperforms traditional LoRA, achieving better results with lower rank configurations.
- NoRA's mechanism activates the dormant tail of the singular value spectrum, preventing rank collapse.
- This advancement has implications for improving model efficiency and effectiveness in various AI applications.
- The results are validated through benchmarks like SlimOrca and MathInstruct, showcasing NoRA's superior spectral efficiency.
Computer Science > Machine Learning arXiv:2602.22911 (cs) [Submitted on 26 Feb 2026] Title:NoRA: Breaking the Linear Ceiling of Low-Rank Adaptation via Manifold Expansion Authors:Hung-Hsuan Chen View a PDF of the paper titled NoRA: Breaking the Linear Ceiling of Low-Rank Adaptation via Manifold Expansion, by Hung-Hsuan Chen View PDF HTML (experimental) Abstract:Low-Rank Adaptation (LoRA) dominates parameter-efficient fine-tuning (PEFT). However, it faces a critical ``linear ceiling'' in complex reasoning tasks: simply increasing the rank yields diminishing returns due to intrinsic linear constraints. We introduce NoRA (Non-linear Rank Adaptation), a weight-level parallel adapter that injects SiLU gating and structural dropout to induce manifold expansion. On the SlimOrca benchmark, NoRA breaks this linear barrier: NoRA remarkably at rank 64 (PPL 3.89) outperforms LoRA at rank 512 (PPL 3.90), demonstrating superior spectral efficiency. This advantage generalizes to mathematical reasoning, where NoRA achieves a perplexity of 1.97 on MathInstruct, significantly surpassing LoRA's saturation point of 2.07. Mechanism analysis via Singular Value Decomposition (SVD) confirms that NoRA activates the dormant tail of the singular value spectrum, effectively preventing the rank collapse observed in linear methods. Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) Cite as: arXiv:2602.22911 [cs.LG] (or arXiv:2602.22911v1 [cs.LG] for ...