[2509.05609] New Insights into Optimal Alignment of Acoustic and Linguistic Representations for Knowledge Transfer in ASR

Computer Science > Computation and Language arXiv:2509.05609 (cs) [Submitted on 6 Sep 2025 (v1), last revised 5 Mar 2026 (this version, v2)] Title:New Insights into Optimal Alignment of Acoustic and Linguistic Representations for Knowledge Transfer in ASR Authors:Xugang Lu, Peng Shen, Hisashi Kawai View a PDF of the paper titled New Insights into Optimal Alignment of Acoustic and Linguistic Representations for Knowledge Transfer in ASR, by Xugang Lu and 2 other authors View PDF HTML (experimental) Abstract:Aligning acoustic and linguistic representations is a central challenge to bridge the pre-trained models in knowledge transfer for automatic speech recognition (ASR). This alignment is inherently structured and asymmetric: while multiple consecutive acoustic frames typically correspond to a single linguistic token (many-to-one), certain acoustic transition regions may relate to multiple adjacent tokens (one-to-many). Moreover, acoustic sequences often include frames with no linguistic counterpart, such as background noise or silence may lead to imbalanced matching conditions. In this work, we take a new insight to regard alignment and matching as a detection problem, where the goal is to identify meaningful correspondences with high precision and recall ensuring full coverage of linguistic tokens while flexibly handling redundant or noisy acoustic frames in transferring linguistic knowledge for ASR. Based on this new insight, we propose an unbalanced optimal transport-ba...