![[2505.03801] Large Language Model Compression with Global Rank and Sparsity Optimization](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2505.03801] Large Language Model Compression with Global Rank and Sparsity Optimization
Summary
This paper presents a novel two-stage method for compressing large language models (LLMs) by optimizing global rank and sparsity, addressing key challenges in performance and resource allocation.
Why It Matters
As large language models grow in size and complexity, efficient compression techniques are essential for practical deployment. This research provides a significant advancement in model optimization, potentially leading to more accessible AI applications and reduced computational costs.
Key Takeaways
- Introduces a two-stage method for LLM compression focusing on global rank and sparsity.
- Addresses challenges in low-rank and sparse matrix interactions and layer-wise weight allocation.
- Utilizes robust principal component analysis to optimize weight matrices effectively.
- Demonstrates superior performance compared to existing sparsification techniques.
- Highlights the importance of redundancy detection across different model layers.
Computer Science > Machine Learning arXiv:2505.03801 (cs) [Submitted on 2 May 2025 (v1), last revised 25 Feb 2026 (this version, v2)] Title:Large Language Model Compression with Global Rank and Sparsity Optimization Authors:Changhai Zhou, Qian Qiao, Yuhua Zhou, Yuxin Wu, Shichao Weng, Weizhong Zhang, Cheng Jin View a PDF of the paper titled Large Language Model Compression with Global Rank and Sparsity Optimization, by Changhai Zhou and 6 other authors View PDF HTML (experimental) Abstract:Low-rank and sparse composite approximation is a natural idea to compress Large Language Models (LLMs). However, such an idea faces two primary challenges that adversely affect the performance of existing methods. The first challenge relates to the interaction and cooperation between low-rank and sparse matrices, while the second involves determining weight allocation across different layers, as redundancy varies considerably among them. To address these challenges, we propose a novel two-stage LLM compression method with the capability of global resource allocation for rank and sparsity. It is noteworthy that the overall optimization space is vast, making comprehensive optimization computationally prohibitive. Therefore, to reduce the optimization space, our first stage utilizes robust principal component analysis to decompose the weight matrices of LLMs into low-rank and sparse components, which span the low dimensional and sparse spaces containing the resultant low-rank and sparse mat...
