![[2602.20492] Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.20492] Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA
Summary
This paper presents a novel approach to decentralized federated learning for multi-task large language model fine-tuning, addressing key challenges in communication and knowledge representation.
Why It Matters
As the demand for efficient and effective machine learning models grows, this research offers significant advancements in federated learning. By tackling issues like knowledge forgetting and communication inefficiencies, it enhances the capability of mobile devices to collaboratively fine-tune models without compromising performance, which is crucial for applications in diverse environments.
Key Takeaways
- Proposes a sparse-and-orthogonal LoRA to mitigate knowledge conflicts during fine-tuning.
- Introduces a cluster-based topology design to improve multi-task performance.
- Presents an implicit mixture of experts mechanism to manage incompatible knowledge during inference.
- Demonstrates a 73% reduction in communication resource consumption compared to traditional methods.
- Achieves a 5% performance improvement over existing LoRA techniques.
Computer Science > Machine Learning arXiv:2602.20492 (cs) [Submitted on 24 Feb 2026] Title:Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA Authors:Nuocheng Yang, Sihua Wang, Ouwen Huan, Mingzhe Chen, Tony Q. S. Quek, Changchuan Yin View a PDF of the paper titled Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA, by Nuocheng Yang and 5 other authors View PDF HTML (experimental) Abstract:Decentralized federated learning (DFL) based on low-rank adaptation (LoRA) enables mobile devices with multi-task datasets to collaboratively fine-tune a large language model (LLM) by exchanging locally updated parameters with a subset of neighboring devices via wireless connections for knowledge this http URL, directly aggregating parameters fine-tuned on heterogeneous datasets induces three primary issues across the DFL life-cycle: (i) \textit{catastrophic knowledge forgetting during fine-tuning process}, arising from conflicting update directions caused by data heterogeneity; (ii) \textit{inefficient communication and convergence during model aggregation process}, due to bandwidth-intensive redundant model transmissions; and (iii) \textit{multi-task knowledge interference during inference process}, resulting from incompatible knowledge representations coexistence during inference. To address these issues in a fully decentralized scenario, we first propose a sparse-and-orthogonal LoRA that ensures orthogonality between model updates t...