![[2505.02819] ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2505.02819] ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization
Summary
The paper presents ReplaceMe, a novel method for network simplification that utilizes depth pruning and transformer block linearization, achieving significant performance retention without retraining.
Why It Matters
As transformer models become increasingly complex, methods like ReplaceMe are crucial for optimizing their efficiency. This approach allows for substantial model compression while maintaining performance, making it relevant for developers and researchers focused on enhancing AI model deployment and resource management.
Key Takeaways
- ReplaceMe simplifies transformer networks by replacing blocks with linear operations.
- The method requires only a small calibration dataset, eliminating extensive retraining.
- Achieves up to 25% pruning while retaining approximately 90% of original performance.
- Provides an open-source library for implementing this technique.
- Outperforms existing training-free pruning methods and competes with state-of-the-art techniques.
Computer Science > Computation and Language arXiv:2505.02819 (cs) [Submitted on 5 May 2025 (v1), last revised 19 Feb 2026 (this version, v4)] Title:ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization Authors:Dmitriy Shopkhoev, Ammar Ali, Magauiya Zhussip, Valentin Malykh, Stamatios Lefkimmiatis, Nikos Komodakis, Sergey Zagoruyko View a PDF of the paper titled ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization, by Dmitriy Shopkhoev and 6 other authors View PDF HTML (experimental) Abstract:We introduce ReplaceMe, a generalized training-free depth pruning method that effectively replaces transformer blocks with a linear operation, while maintaining high performance for low compression ratios. In contrast to conventional pruning approaches that require additional training or fine-tuning, our approach requires only a small calibration dataset that is used to estimate a linear transformation, which approximates the pruned blocks. The estimated linear mapping can be seamlessly merged with the remaining transformer blocks, eliminating the need for any additional network parameters. Our experiments show that ReplaceMe consistently outperforms other training-free approaches and remains highly competitive with state-of-the-art pruning methods that involve extensive retraining/fine-tuning and architectural modifications. Applied to several large language models (LLMs), ReplaceMe achieves up to 25\% pruning while ...
