![[2602.20981] Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.20981] Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models
Summary
This paper presents MMHNet, a novel multimodal hierarchical network that enhances video-to-audio generation by enabling models to generalize from short to long audio outputs, achieving significant improvements in performance.
Why It Matters
As the demand for high-quality audio generation from video content increases, this research addresses a critical challenge in the field of multimodal AI. By demonstrating that models can effectively generalize from short to long audio, it opens new avenues for applications in media production, accessibility, and entertainment.
Key Takeaways
- MMHNet significantly improves long audio generation capabilities.
- The model can generalize from short training instances to longer audio outputs.
- Achieves state-of-the-art results in video-to-audio benchmarks, outperforming previous methods.
Computer Science > Computer Vision and Pattern Recognition arXiv:2602.20981 (cs) [Submitted on 24 Feb 2026] Title:Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models Authors:Christian Simon, MAsato Ishii, Wei-Yao Wang, Koichi Saito, Akio Hayakawa, Dongseok Shim, Zhi Zhong, Shuyang Cui, Shusuke Takahashi, Takashi Shibuya, Yuki Mitsufuji View a PDF of the paper titled Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models, by Christian Simon and 10 other authors View PDF HTML (experimental) Abstract:Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkabl...
