[2603.01195] VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning

Computer Science > Computer Vision and Pattern Recognition arXiv:2603.01195 (cs) [Submitted on 1 Mar 2026] Title:VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning Authors:Mingkang Dong, Hongyi Cai, Jie Li, Sifan Zhou, Bin Ren, Kunyu Peng, Yuqian Fu View a PDF of the paper titled VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning, by Mingkang Dong and 6 other authors View PDF HTML (experimental) Abstract:The effectiveness of multimodal instruction tuning depends not only on dataset scale, but critically on whether training samples genuinely require visual reasoning. However, existing instruction datasets often contain a substantial portion of visually redundant samples (solvable from text alone), as well as multimodally misaligned supervision that can degrade learning. To address this, we propose VisNec (Visual Necessity Score), a principled data selection framework that measures the marginal contribution of visual input during instruction tuning. By comparing predictive loss with and without visual context, VisNec identifies whether a training instance is vision-critical, redundant, or misaligned. To preserve task diversity, we combine VisNec with semantic clustering and select high-necessity samples within each cluster. Across 10 downstream benchmarks, training on only 15% of the LLaVA-665K dataset selected by VisNec achieves 100.2% of full-data performance. On the smaller Vision-Flan-186K dataset, our sel...