[2603.28026] When Choices Become Priors: Contrastive Decoding for Scientific Figure Multiple-Choice QA

Computer Science > Artificial Intelligence arXiv:2603.28026 (cs) [Submitted on 30 Mar 2026] Title:When Choices Become Priors: Contrastive Decoding for Scientific Figure Multiple-Choice QA Authors:Taeyun Roh, Eun-yeong Jo, Wonjune Jang, Jaewoo Kang View a PDF of the paper titled When Choices Become Priors: Contrastive Decoding for Scientific Figure Multiple-Choice QA, by Taeyun Roh and 3 other authors View PDF HTML (experimental) Abstract:Scientific figure multiple-choice question answering (MCQA) requires models to reason over diverse visual evidence, ranging from charts and multipanel figures to microscopy and biomedical images. However, this setting suffers from a distinctive bias: answer choices themselves can act as priors, steering multimodal models toward scientifically plausible options even when the figure supports a different answer. We investigate this failure mode through a simple question: what if decoding explicitly discounts what the model would prefer from text alone, so as to favor figure-grounded evidence? To this end, we propose SCICON, a training-free decoding method that scores each candidate by subtracting a text-only option score from its image-conditioned counterpart. Unlike prior contrastive decoding approaches that mitigate hallucinations by contrasting original inputs with distorted images or perturbed instructions, SCICON directly targets the choice-induced prior encoded in candidate text. Across three scientific figure QA benchmarks and three mo...