[2603.00563] Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion

Computer Science > Sound arXiv:2603.00563 (cs) [Submitted on 28 Feb 2026] Title:Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion Authors:Sen Zhang, Jianguo Wei, Wenhuan Lu, Xianghu Yue, Wei Li, Qiang Li, Pengcheng Zhao, Ming Cai, Luo Si View a PDF of the paper titled Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion, by Sen Zhang and 8 other authors View PDF HTML (experimental) Abstract:The Transformer-based Whisper model has achieved state-of-the-art performance in Automatic Speech Recognition (ASR). However, its Multi-Head Attention (MHA) mechanism results in significant GPU memory consumption due to the linearly growing Key-Value (KV) cache usage, which is problematic for many applications especially with long-form audio. To address this, we introduce Whisper-MLA, a novel architecture that incorporates Multi-Head Latent Attention (MLA) into the Whisper model. Specifically, we adapt MLA for Whisper's absolute positional embeddings and systematically investigate its application across encoder self-attention, decoder self-attention, and cross-attention modules. Empirical results indicate that applying MLA exclusively to decoder self-attention yields the desired balance between performance and memory efficiency. Our proposed approach allows conversion of a pretrained Whisper model to Whisper-MLA with minimal fine-tuning. Extensive experiments on the LibriSpeech benchmark validate the effectiveness of...