![[2602.21652] Sparsity Induction for Accurate Post-Training Pruning of Large Language Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.21652] Sparsity Induction for Accurate Post-Training Pruning of Large Language Models
Summary
The paper presents a novel method called Sparsity Induction for enhancing post-training pruning of large language models, addressing challenges in computational efficiency and model performance.
Why It Matters
As large language models grow in size, their computational and memory demands increase, making efficient pruning essential. This research proposes a method that not only improves sparsity but also maintains model performance, which is crucial for deploying these models in resource-constrained environments.
Key Takeaways
- Sparsity Induction enhances model sparsity at both distribution and feature levels.
- The method avoids performance degradation during pruning by using mathematically equivalent scaling transformations.
- Experiments show superior pruning performance compared to existing methods across various model architectures.
Computer Science > Computation and Language arXiv:2602.21652 (cs) [Submitted on 25 Feb 2026] Title:Sparsity Induction for Accurate Post-Training Pruning of Large Language Models Authors:Minhao Jiang, Zhikai Li, Xuewen Liu, Jing Zhang, Mengjuan Chen, Qingyi Gu View a PDF of the paper titled Sparsity Induction for Accurate Post-Training Pruning of Large Language Models, by Minhao Jiang and 5 other authors View PDF HTML (experimental) Abstract:Large language models have demonstrated capabilities in text generation, while their increasing parameter scales present challenges in computational and memory efficiency. Post-training sparsity (PTS), which reduces model cost by removing weights from dense networks, is an effective approach. However, native dense matrices lack high sparsity, making existing approaches that directly remove weights disrupt model states, resulting in unsatisfactory performance recovery even with post-tuning. We propose Sparsity Induction, which promotes models toward higher sparsity at both distribution and feature levels before pruning, to push the limits of PTS. At the distribution level, we enhance distributional sparsity through mathematically equivalent scaling transformations, which are fully absorbable and incur no extra parameters or inference-time overhead. At the feature level, we introduce Spectral Norm Loss to promote feature sparsity from a low-rank perspective. Experiments across diverse model architectures and tasks demonstrate that our met...
