![[2505.19427] WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2505.19427] WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference
Summary
The paper introduces WINA, a novel framework for efficient inference in large language models (LLMs) that optimally combines hidden state magnitudes and weight matrix norms for sparse activation.
Why It Matters
As large language models become increasingly resource-intensive, optimizing their inference processes is crucial. WINA offers a training-free method that enhances performance while maintaining efficiency, potentially setting a new standard in the field.
Key Takeaways
- WINA provides a training-free sparse activation method for LLMs.
- It combines hidden state magnitudes with weight matrix norms for improved accuracy.
- Empirical results show WINA outperforms existing methods by up to 2.94% at the same sparsity levels.
Computer Science > Machine Learning arXiv:2505.19427 (cs) [Submitted on 26 May 2025 (v1), last revised 18 Feb 2026 (this version, v2)] Title:WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference Authors:Sihan Chen, Dan Zhao, Jongwoo Ko, Colby Banbury, Huiping Zhuang, Luming Liang, Pashmina Cameron, Tianyi Chen View a PDF of the paper titled WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference, by Sihan Chen and 7 other authors View PDF Abstract:The growing computational demands of large language models (LLMs) make efficient inference and activation strategies increasingly critical. While recent approaches, such as Mixture-of-Experts (MoE), leverage selective activation but require specialized training, training-free sparse activation methods offer broader applicability and superior resource efficiency through their plug-and-play design. However, many existing methods rely solely on hidden state magnitudes to determine activation, resulting in high approximation errors and suboptimal inference accuracy. To address these limitations, we propose WINA (Weight Informed Neuron Activation), a novel, simple, and training-free sparse activation framework that jointly considers hidden state magnitudes and the column-wise $\ell_2$-norms of weight matrices. We show that this leads to a sparsification strategy that obtains optimal approximation error bounds with theoretical guarantees tighter than existing techniques....