[2603.05231] Boosting ASR Robustness via Test-Time Reinforcement Learning with Audio-Text Semantic Rewards

Computer Science > Sound arXiv:2603.05231 (cs) [Submitted on 5 Mar 2026] Title:Boosting ASR Robustness via Test-Time Reinforcement Learning with Audio-Text Semantic Rewards Authors:Linghan Fang, Tianxin Xie, Li Liu View a PDF of the paper titled Boosting ASR Robustness via Test-Time Reinforcement Learning with Audio-Text Semantic Rewards, by Linghan Fang and 1 other authors View PDF HTML (experimental) Abstract:Recently, Automatic Speech Recognition (ASR) systems (e.g., Whisper) have achieved remarkable accuracy improvements but remain highly sensitive to real-world unseen data (data with large distribution shifts), including noisy environments and diverse accents. To address this issue, test-time adaptation (TTA) has shown great potential in improving the model adaptability at inference time without ground-truth labels, and existing TTA methods often rely on pseudo-labeling or entropy minimization. However, by treating model confidence as a learning signal, these methods may reinforce high-confidence errors, leading to confirmation bias that undermines adaptation. To overcome these limitations, we present ASR-TRA, a novel Test-time Reinforcement Adaptation framework inspired by causal intervention. More precisely, our method introduces a learnable decoder prompt and utilizes temperature-controlled stochastic decoding to generate diverse transcription candidates. These are scored by a reward model that measures audio-text semantic alignment, and the resulting feedback is u...