![[2501.00339] GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2501.00339] GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression
Summary
The paper introduces GRASP, a novel framework for model compression that replaces redundant layers in large language models with adaptive singular parameters, achieving efficient compression while maintaining performance.
Why It Matters
As large language models become increasingly complex and resource-intensive, efficient model compression techniques like GRASP are essential for reducing computational costs and improving deployment feasibility without sacrificing performance. This research contributes to the ongoing efforts in optimizing AI models for practical applications.
Key Takeaways
- GRASP identifies and retains critical singular components to enhance model efficiency.
- The framework achieves up to 20% compression while maintaining 90% of the original model's performance.
- Gradient-based attribution is used for sensitivity-aware parameter retention, improving upon traditional layer pruning methods.
Computer Science > Computation and Language arXiv:2501.00339 (cs) [Submitted on 31 Dec 2024 (v1), last revised 22 Feb 2026 (this version, v4)] Title:GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression Authors:Kainan Liu, Yong Zhang, Ning Cheng, Zhitao Li, Shaojun Wang, Jing Xiao View a PDF of the paper titled GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression, by Kainan Liu and 5 other authors View PDF HTML (experimental) Abstract:Recent studies have demonstrated that many layers are functionally redundant in large language models (LLMs), enabling model compression by removing these layers to reduce inference cost. While such approaches can improve efficiency, indiscriminate layer pruning often results in significant performance degradation. In this paper, we propose GRASP (Gradient-based Retention of Adaptive Singular Parameters), a novel compression framework that mitigates this issue by preserving sensitivity-aware singular values. Unlike direct layer pruning, GRASP leverages gradient-based attribution on a small calibration dataset to adaptively identify and retain critical singular components. By replacing redundant layers with only a minimal set of parameters, GRASP achieves efficient compression while maintaining strong performance with minimal overhead. Experiments across multiple LLMs show that GRASP consistently outperforms existing compression methods, achieving 90% of t...
