[2604.04988] Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression

Computer Science > Machine Learning arXiv:2604.04988 (cs) [Submitted on 5 Apr 2026] Title:Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression Authors:Longsheng Zhou, Yu Shen View a PDF of the paper titled Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression, by Longsheng Zhou and 1 other authors View PDF HTML (experimental) Abstract:Modern deployment often requires trading accuracy for efficiency under tight CPU and memory constraints, yet common compression proxies such as parameter count or FLOPs do not reliably predict wall-clock inference time. In particular, unstructured sparsity can reduce model storage while failing to accelerate (and sometimes slightly slowing down) standard CPU execution due to irregular memory access and sparse kernel overhead. Motivated by this gap between compression and acceleration, we study a practical, ordered pipeline that targets measured latency by combining three widely used techniques: unstructured pruning, INT8 quantization-aware training (QAT), and knowledge distillation (KD). Empirically, INT8 QAT provides the dominant runtime benefit, while pruning mainly acts as a capacity-reduction pre-conditioner that improves the robustness of subsequent low-precision optimization; KD, applied last, recovers accuracy within the already constrained sparse INT8 regime without changing the deployment form. We evaluate on CIFAR-10/100 using three backbones (ResNet-18, WRN-28-10, and V...