![[2602.14490] Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.14490] Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts
Summary
This paper introduces Mixture of Space (MoS), a novel framework for parameter-efficient fine-tuning of large language models (LLMs) that utilizes multiple geometric spaces to enhance model performance and adaptability.
Why It Matters
As large language models continue to evolve, the need for efficient fine-tuning methods becomes critical. MoS offers a significant advancement by allowing models to leverage various geometric representations, potentially improving their ability to handle complex language tasks and enhancing overall performance.
Key Takeaways
- MoS framework enables simultaneous use of multiple geometric spaces for richer representations.
- MoSLoRA extends Low-Rank Adaptation with heterogeneous geometric experts for improved model adaptability.
- Empirical results show significant performance improvements on various benchmarks, including MATH500 and MAWPS.
- The lightweight routing mechanism addresses computational overhead associated with manifold switching.
- Curvature optimization is shown to impact training stability and model performance positively.
Computer Science > Machine Learning arXiv:2602.14490 (cs) [Submitted on 16 Feb 2026] Title:Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts Authors:Buze Zhang, Jinkai Tao, Zilang Zeng, Neil He, Ali Maatouk, Menglin Yang, Rex Ying View a PDF of the paper titled Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts, by Buze Zhang and 6 other authors View PDF HTML (experimental) Abstract:Large Language Models (LLMs) have achieved remarkable progress, with Parameter-Efficient Fine-Tuning (PEFT) emerging as a key technique for downstream task adaptation. However, existing PEFT methods mainly operate in Euclidean space, fundamentally limiting their capacity to capture complex geometric structures inherent in language data. While alternative geometric spaces, like hyperbolic geometries for hierarchical data and spherical manifolds for circular patterns, offer theoretical advantages, forcing representations into a single manifold type ultimately limits expressiveness, even when curvature parameters are learnable. To address this, we propose Mixture of Space (MoS), a unified framework that leverages multiple geometric spaces simultaneously to learn richer, curvature-aware representations. Building on this scheme, we develop MoSLoRA, which extends Low-Rank Adaptation (LoRA) with heterogeneous geometric experts, enabling models to dynamically select or combine appropriate geometric spaces based on input context. Furthermore, to address the computa...
