[2603.02731] Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs

Computer Science > Machine Learning arXiv:2603.02731 (cs) [Submitted on 3 Mar 2026] Title:Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs Authors:Wuyue Zhang, Chongdong Huang, Chunbo You, Cheng Gu, Fengjuan Wang, Mou Sun View a PDF of the paper titled Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs, by Wuyue Zhang and 5 other authors View PDF HTML (experimental) Abstract:Training large-scale Mixture-of-Experts (MoE) models is bottlenecked by activation memory and expert-parallel communication, yet FP4 training remains impractical on Hopper-class GPUs without native MXFP4 or NVFP4 support. In this work, we present a training recipe that enables MXFP4 efficiency for MoE models on Hopper architectures without native 4-bit computation support. A central challenge is to integrate FP4 into an existing BF16/FP8 hybrid training pipeline without incurring costly precision round-trips (e.g., FP4 $\leftrightarrow$ BF16 $\leftrightarrow$ FP8). We address this challenge by introducing direct FP8-to-FP4 quantization and de-quantization, together with scaling-aware FP4 row-wise to column-wise conversion, enabling FP4 activations and expert-parallel communication with minimal overhead. Core MoE computations are executed in FP8, while activations and expert-parallel communication are compressed using MXFP4, achieving substantial memory and bandwidth savings without degrading convergence. At the 671B parameter scale, our method achieves end-to-end training ...