![[2404.16890] Layer Collapse Can be Induced by Unstructured Pruning](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2404.16890] Layer Collapse Can be Induced by Unstructured Pruning
Summary
This paper explores how unstructured pruning can lead to layer collapse in neural networks, demonstrating that it can effectively reduce model complexity without significant performance loss.
Why It Matters
Understanding the implications of unstructured pruning is crucial for optimizing neural network architectures. This research reveals that pruning can not only reduce parameters but also eliminate entire layers, enhancing model efficiency and performance in various applications.
Key Takeaways
- Unstructured pruning can induce layer collapse in neural networks.
- Magnitude-based pruning reduces neuron entropy, allowing for layer removal.
- The method is validated across CNNs, Vision Transformers, and NLP models.
- Effective layer removal can occur with minimal performance degradation.
- This research contributes to more efficient model design in AI.
Computer Science > Machine Learning arXiv:2404.16890 (cs) [Submitted on 24 Apr 2024 (v1), last revised 23 Feb 2026 (this version, v2)] Title:Layer Collapse Can be Induced by Unstructured Pruning Authors:Zhu Liao, Victor Quétu, Van-Tam Nguyen, Enzo Tartaglione View a PDF of the paper titled Layer Collapse Can be Induced by Unstructured Pruning, by Zhu Liao and 3 other authors View PDF Abstract:Unstructured pruning is a popular compression method for efficiently reducing model parameters. However, while it effectively decreases the number of parameters, it is commonly believed that unstructured pruning cannot shorten the computational critical path, i.e., the maximum number of layers traversed during forward propagation. In this paper, we study when and how unstructured pruning can yield structural effects. For rectifier-activated networks, we introduce the notion of neuron entropy, which quantifies the degree of nonlinearity utilization. We show that magnitude-based pruning naturally lowers this entropy, sometimes down to zero-entropy layers that become linearizable and can thus be removed. Building on this insight, we propose a method that leverages "unstructured" pruning to favor sparsity in low-entropy layers, enabling their complete removal. We validate the phenomenon across CNNs, Vision Transformers, and NLP models: unstructured pruning can induce effective layer removal with little or no performance degradation in over-parameterized networks. Subjects: Machine Learnin...