[2603.20180] Adaptive Greedy Frame Selection for Long Video Understanding

Computer Science > Computer Vision and Pattern Recognition arXiv:2603.20180 (cs) [Submitted on 20 Mar 2026] Title:Adaptive Greedy Frame Selection for Long Video Understanding Authors:Yuning Huang, Fengqing Zhu View a PDF of the paper titled Adaptive Greedy Frame Selection for Long Video Understanding, by Yuning Huang and Fengqing Zhu View PDF HTML (experimental) Abstract:Large vision--language models (VLMs) are increasingly applied to long-video question answering, yet inference is often bottlenecked by the number of input frames and resulting visual tokens. Naive sparse sampling can miss decisive moments, while purely relevance-driven selection frequently collapses onto near-duplicate frames and sacrifices coverage of temporally distant evidence. We propose a question-adaptive greedy frame selection method that jointly optimizes query relevance and semantic representativeness under a fixed frame budget. Our approach constructs a 1~FPS candidate pool (capped at 1000) with exact timestamp alignment, embeds candidates in two complementary spaces (SigLIP for question relevance and DINOv2 for semantic similarity), and selects frames by greedily maximizing a weighted sum of a modular relevance term and a facility-location coverage term. This objective is normalized, monotone, and submodular, yielding a standard (1-1/e) greedy approximation guarantee. To account for question-dependent trade-offs between relevance and coverage, we introduce four preset strategies and a lightweigh...