![[2602.18527] JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.18527] JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments
Summary
The paper presents JAEGER, a framework for joint 3D audio-visual grounding and reasoning, addressing limitations of existing 2D models by integrating RGB-D observations and multi-channel audio for enhanced spatial perception.
Why It Matters
As AI systems increasingly operate in complex physical environments, the ability to accurately perceive and reason in 3D is crucial. JAEGER's approach not only improves spatial reasoning but also sets a new benchmark for future research in audio-visual AI, highlighting the importance of 3D modeling.
Key Takeaways
- JAEGER extends audio-visual large language models to 3D environments.
- Introduces Neural Intensity Vector for improved spatial audio representation.
- Demonstrates superior performance over 2D-centric models in spatial tasks.
- Proposes a new benchmark, SpatialSceneQA, for systematic evaluation.
- Highlights the necessity of 3D modeling in advancing AI capabilities.
Computer Science > Computer Vision and Pattern Recognition arXiv:2602.18527 (cs) [Submitted on 20 Feb 2026] Title:JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments Authors:Zhan Liu, Changli Tang, Yuxin Wang, Zhiyuan Zhu, Youjun Chen, Yiwen Shao, Tianzi Wang, Lei Ke, Zengrui Jin, Chao Zhang View a PDF of the paper titled JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments, by Zhan Liu and 9 other authors View PDF HTML (experimental) Abstract:Current audio-visual large language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB video and monaural audio. This design choice introduces a fundamental dimensionality mismatch that precludes reliable source localization and spatial reasoning in complex 3D environments. We address this limitation by presenting JAEGER, a framework that extends AV-LLMs to 3D space, to enable joint spatial grounding and reasoning through the integration of RGB-D observations and multi-channel first-order ambisonics. A core contribution of our work is the neural intensity vector (Neural IV), a learned spatial audio representation that encodes robust directional cues to enhance direction-of-arrival estimation, even in adverse acoustic scenarios with overlapping sources. To facilitate large-scale training and systematic evaluation, we propose SpatialSceneQA, a benchmark of 61k instruction-tuning samples curated from simulated physical environments. Exte...
