[2603.20307] EARTalking: End-to-end GPT-style Autoregressive Talking Head Synthesis with Frame-wise Control

Computer Science > Computer Vision and Pattern Recognition arXiv:2603.20307 (cs) [Submitted on 19 Mar 2026] Title:EARTalking: End-to-end GPT-style Autoregressive Talking Head Synthesis with Frame-wise Control Authors:Yuzhe Weng, Haotian Wang, Yuanhong Yu, Jun Du, Shan He, Xiaoyan Wu, Haoran Xu View a PDF of the paper titled EARTalking: End-to-end GPT-style Autoregressive Talking Head Synthesis with Frame-wise Control, by Yuzhe Weng and 6 other authors View PDF HTML (experimental) Abstract:Audio-driven talking head generation aims to create vivid and realistic videos from a static portrait and speech. Existing AR-based methods rely on intermediate facial representations, which limit their expressiveness and realism. Meanwhile, diffusion-based methods generate clip-by-clip, lacking fine-grained control and causing inherent latency due to overall denoising across the window. To address these limitations, we propose EARTalking, a novel end-to-end, GPT-style autoregressive model for interactive audio-driven talking head generation. Our method introduces a novel frame-by-frame, in-context, audio-driven streaming generation paradigm. For inherently supporting variable-length video generation with identity consistency, we propose the Sink Frame Window Attention (SFA) mechanism. Furthermore, to avoid the complex, separate networks that prior works required for diverse control signals, we propose a streaming Frame Condition In-Context (FCIC) scheme. This scheme efficiently injects d...