[2603.27958] CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

Computer Science > Artificial Intelligence arXiv:2603.27958 (cs) [Submitted on 30 Mar 2026] Title:CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs Authors:Yongkang Du, Xiaohan Zou, Minhao Cheng, Lu Lin View a PDF of the paper titled CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs, by Yongkang Du and 3 other authors View PDF HTML (experimental) Abstract:Analogical reasoning tests a fundamental aspect of human cognition: mapping the relation from one pair of objects to another. Existing evaluations of this ability in multimodal large language models (MLLMs) overlook the ability to compose rules from multiple sources, a critical component of higher-order intelligence. To close this gap, we introduce CARV (Compositional Analogical Reasoning in Vision), a novel task together with a 5,500-sample dataset as the first diagnostic benchmark. We extend the analogy from a single pair to multiple pairs, which requires MLLMs to extract symbolic rules from each pair and compose new transformations. Evaluation on the state-of-the-art MLLMs reveals a striking performance gap: even Gemini-2.5 Pro achieving only 40.4% accuracy, far below human-level performance of 100%. Diagnostic analysis shows two consistent failure modes: (1) decomposing visual changes into symbolic rules, and (2) maintaining robustness under diverse or complex settings, highlighting the limitations of current MLLMs on this task. Subjects: Artifi...