[2604.06327] A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speech

Computer Science > Sound arXiv:2604.06327 (cs) [Submitted on 7 Apr 2026] Title:A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speech Authors:Jia-Hong Huang, Seulgi Kim, Yi Chieh Liu, Yixian Shen, Hongyi Zhu, Prayag Tiwari, Stevan Rudinac, Evangelos Kanoulas View a PDF of the paper titled A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speech, by Jia-Hong Huang and 7 other authors View PDF HTML (experimental) Abstract:Recent diffusion-based text-to-speech (TTS) models achieve high naturalness and expressiveness, yet often suffer from speaker drift, a subtle, gradual shift in perceived speaker identity within a single utterance. This underexplored phenomenon undermines the coherence of synthetic speech, especially in long-form or interactive settings. We introduce the first automatic framework for detecting speaker drift by formulating it as a binary classification task over utterance-level speaker consistency. Our method computes cosine similarity across overlapping segments of synthesized speech and prompts large language models (LLMs) with structured representations to assess drift. We provide theoretical guarantees for cosine-based drift detection and demonstrate that speaker embeddings exhibit meaningful geometric clustering on the unit sphere. To support evaluation, we construct a high-quality synthetic benchmark with human-validated speaker drift annotations. Experiments with multiple state-of-the-art LLMs confirm the via...