[2601.14001] Cross-Sensory Brain Passage Retrieval: Scaling Beyond Visual to Audio

Computer Science > Information Retrieval arXiv:2601.14001 (cs) [Submitted on 20 Jan 2026 (v1), last revised 24 Mar 2026 (this version, v2)] Title:Cross-Sensory Brain Passage Retrieval: Scaling Beyond Visual to Audio Authors:Niall McGuire, Yashar Moshfeghi View a PDF of the paper titled Cross-Sensory Brain Passage Retrieval: Scaling Beyond Visual to Audio, by Niall McGuire and Yashar Moshfeghi View PDF HTML (experimental) Abstract:Query formulation from internal information needs remains fundamentally challenging across all Information Retrieval paradigms due to cognitive complexity and physical impairments. Brain Passage Retrieval (BPR) addresses this by directly mapping EEG signals to passage representations without intermediate text translation. However, existing BPR research exclusively uses visual stimuli, leaving critical questions unanswered: Can auditory EEG enable effective retrieval for voice-based interfaces and visually impaired users? Can training on combined EEG datasets from different sensory modalities improve performance despite severe data scarcity? We present the first systematic investigation of auditory EEG for BPR and evaluate cross-sensory training benefits. Using dual encoder architectures with four pooling strategies (CLS, mean, max, multi-vector), we conduct controlled experiments comparing auditory-only, visual-only, and combined training on the Alice (auditory) and Nieuwland (visual) datasets. Results demonstrate that auditory EEG consistently ou...