![[2507.03262] Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2507.03262] Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders
Summary
This article investigates redundancy in multimodal large language models (MLLMs) with multiple vision encoders, revealing that more encoders do not always lead to better performance.
Why It Matters
Understanding redundancy in MLLMs is crucial for optimizing model efficiency and performance. This research challenges the prevailing assumption that adding more encoders enhances capabilities, providing insights for future model design and resource allocation.
Key Takeaways
- Redundancy in MLLMs can lead to performance improvements when certain encoders are masked.
- The Conditional Utilization Rate (CUR) and Information Gap (IG) metrics help quantify encoder contributions.
- Specialization in tasks like OCR shows that a single encoder can dominate performance.
- High redundancy is observed in general VQA tasks, indicating encoders are often interchangeable.
- Masking specific encoders can yield significant accuracy boosts, challenging the 'more is better' paradigm.
Computer Science > Computer Vision and Pattern Recognition arXiv:2507.03262 (cs) [Submitted on 4 Jul 2025 (v1), last revised 13 Feb 2026 (this version, v4)] Title:Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders Authors:Yizhou Wang, Song Mao, Yang Chen, Yufan Shen, Yinqiao Yan, Pinlong Cai, Ding Wang, Guohang Yan, Zhi Yu, Xuming Hu, Botian Shi View a PDF of the paper titled Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders, by Yizhou Wang and 10 other authors View PDF Abstract:Recent multimodal large language models (MLLMs) increasingly integrate multiple vision encoders to improve performance on various benchmarks, assuming that diverse pretraining objectives yield complementary visual signals. However, we show this assumption often fails in practice. Through systematic encoder masking across representative multi encoder MLLMs, we find that performance typically degrades gracefully, and sometimes even improves, when selected encoders are masked, revealing pervasive encoder redundancy. To quantify this effect, we introduce two principled metrics: the Conditional Utilization Rate (CUR), which measures an encoder s marginal contribution in the presence of others, and the Information Gap (IG), which captures heterogeneity in encoder utility within a model. Using these tools, we observe: (i) strong specialization on tasks like OCR and Chart, where a single encoder can dominate with a CUR greater...
