[2603.26775] Learning to Select Visual In-Context Demonstrations

Computer Science > Machine Learning arXiv:2603.26775 (cs) [Submitted on 24 Mar 2026] Title:Learning to Select Visual In-Context Demonstrations Authors:Eugene Lee, Yu-Chi Lin, Jiajie Diao View a PDF of the paper titled Learning to Select Visual In-Context Demonstrations, by Eugene Lee and 2 other authors View PDF HTML (experimental) Abstract:Multimodal Large Language Models (MLLMs) adapt to visual tasks via in-context learning (ICL), which relies heavily on demonstration quality. The dominant demonstration selection strategy is unsupervised k-Nearest Neighbor (kNN) search. While simple, this similarity-first approach is sub-optimal for complex factual regression tasks; it selects redundant examples that fail to capture the task's full output range. We reframe selection as a sequential decision-making problem and introduce Learning to Select Demonstrations (LSD), training a Reinforcement Learning agent to construct optimal demonstration sets. Using a Dueling DQN with a query-centric Transformer Decoder, our agent learns a policy that maximizes MLLM downstream performance. Evaluating across five visual regression benchmarks, we uncover a crucial dichotomy: while kNN remains optimal for subjective preference tasks, LSD significantly outperforms baselines on objective, factual regression tasks. By balancing visual relevance with diversity, LSD better defines regression boundaries, illuminating when learned selection is strictly necessary for visual ICL. Comments: Subjects: Mach...