![[2602.17095] FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.17095] FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment
Summary
The paper presents FLoRG, a federated fine-tuning framework that utilizes low-rank Gram matrices and Procrustes alignment to enhance the efficiency and accuracy of large language model adaptation while minimizing communication overhead.
Why It Matters
FLoRG addresses significant challenges in federated learning, particularly in the context of fine-tuning large language models. By reducing aggregation errors and communication costs, it enables more efficient collaboration across distributed clients, which is crucial for maintaining data privacy and improving model performance in real-world applications.
Key Takeaways
- FLoRG employs a single low-rank matrix for federated fine-tuning, reducing aggregation errors.
- The introduction of Procrustes alignment minimizes decomposition drift during updates.
- FLoRG outperforms five state-of-the-art baseline methods in downstream task accuracy.
- The framework significantly reduces communication overhead by up to 2041 times.
- Theoretical analysis confirms improved convergence bounds with Procrustes alignment.
Computer Science > Machine Learning arXiv:2602.17095 (cs) [Submitted on 19 Feb 2026] Title:FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment Authors:Chuiyang Meng, Ming Tang, Vincent W.S. Wong View a PDF of the paper titled FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment, by Chuiyang Meng and 2 other authors View PDF HTML (experimental) Abstract:Parameter-efficient fine-tuning techniques such as low-rank adaptation (LoRA) enable large language models (LLMs) to adapt to downstream tasks efficiently. Federated learning (FL) further facilitates this process by enabling collaborative fine-tuning across distributed clients without sharing private data. However, the use of two separate low-rank matrices in LoRA for federated fine-tuning introduces two types of challenges. The first challenge arises from the error induced by separately aggregating those two low-rank matrices. The second challenge occurs even when the product of two low-rank matrices is aggregated. The server needs to recover factors via matrix decomposition, which is non-unique and can introduce decomposition drift. To tackle the aforementioned challenges, we propose FLoRG, a federated fine-tuning framework which employs a single low-rank matrix for fine-tuning and aggregates its Gram matrix (i.e., the matrix of inner products of its column vectors), eliminating the aggregation error while also reducing the communication overhead. FLoRG minimizes t...