![[2602.14432] S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.14432] S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations
Summary
The paper introduces Selective Spectral Decay (S2D), a method to improve quantization in neural networks by addressing activation outliers, enhancing model efficiency without sacrificing accuracy.
Why It Matters
With the increasing scale of transformer models, managing activation outliers is crucial for maintaining accuracy during quantization. S2D offers a novel approach to this problem, potentially impacting deployment efficiency across various AI applications.
Key Takeaways
- S2D reduces activation outliers in large-scale transformer models.
- The method improves post-training quantization accuracy by up to 7% on ImageNet.
- S2D generalizes well across different downstream tasks and vision-language models.
- The approach allows for scaling models without compromising deployment efficiency.
- Empirical studies link activation outliers to dominant singular values of weights.
Computer Science > Machine Learning arXiv:2602.14432 (cs) [Submitted on 16 Feb 2026] Title:S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations Authors:Arnav Chavan, Nahush Lele, Udbhav Bamba, Sankalp Dayal, Aditi Raghunathan, Deepak Gupta View a PDF of the paper titled S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations, by Arnav Chavan and Nahush Lele and Udbhav Bamba and Sankalp Dayal and Aditi Raghunathan and Deepak Gupta View PDF HTML (experimental) Abstract:Activation outliers in large-scale transformer models pose a fundamental challenge to model quantization, creating excessively large ranges that cause severe accuracy drops during quantization. We empirically observe that outlier severity intensifies with pre-training scale (e.g., progressing from CLIP to the more extensively trained SigLIP and SigLIP2). Through theoretical analysis as well as empirical correlation studies, we establish the direct link between these activation outliers and dominant singular values of the weights. Building on this insight, we propose Selective Spectral Decay ($S^2D$), a geometrically-principled conditioning method that surgically regularizes only the weight components corresponding to the largest singular values during fine-tuning. Through extensive experiments, we demonstrate that $S^2D$ significantly reduces activation outliers and produces well-conditioned representations that are inherently quantizatio...
