![[2602.12635] Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.12635] Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats
Summary
This article evaluates HiFloat formats for low-bit inference on Ascend NPUs, highlighting their efficiency and compatibility with state-of-the-art quantization frameworks.
Why It Matters
As large language models (LLMs) continue to grow, optimizing their performance through low-bit floating-point formats is crucial. This research provides insights into how HiFloat formats can enhance inference efficiency, which is significant for developers and researchers working on AI and machine learning applications.
Key Takeaways
- INT8 is effective for narrow-range data, while HiFloat formats excel with high-variance data.
- HiF4's hierarchical scaling prevents accuracy collapse in 4-bit regimes.
- HiFloat formats are compatible with leading post-training quantization frameworks.
- Low-bit inference can significantly enhance efficiency in LLM applications.
- The evaluation provides a pathway for optimizing AI model performance on NPUs.
Computer Science > Computation and Language arXiv:2602.12635 (cs) [Submitted on 13 Feb 2026] Title:Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats Authors:Pengxiang Zhao, Hui-Ling Zhen, Xing Li, Han Bao, Weizhe Lin, Zhiyuan Yang, Ziwei Yu, Xin Wang, Mingxuan Yuan, Xianzhi Yu, Zhenhua Dong View a PDF of the paper titled Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats, by Pengxiang Zhao and 10 other authors View PDF HTML (experimental) Abstract:As LLMs scale, low-bit floating-point formats like MXFP and NVFP4 offer new opportunities for precision and efficiency. In this work, we evaluate HiFloat (HiF8 and HiF4), a family of formats tailored for Ascend NPUs. Through rigorous comparison across weight-activation and KV-cache tasks, we provide three key insights: (1) INT8 suits narrow-range data, while floating-point formats excel with high-variance data; (2) in 4-bit regimes, HiF4's hierarchical scaling prevents the accuracy collapse seen in integer formats; and (3) HiFloat is fully compatible with state-of-the-art post-training quantization frameworks. Overall, HiFloat provides a solution for high-efficiency LLM inference on NPUs. Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) Cite as: arXiv:2602.12635 [cs.CL] (or arXiv:2602.12635v1 [cs.CL] for this version) https://doi.org/10.48550/arXiv.2602.12635 Focus to learn more arXiv-issued DO...
