[2603.22368] When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations

Computer Science > Computer Vision and Pattern Recognition arXiv:2603.22368 (cs) [Submitted on 23 Mar 2026] Title:When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations Authors:Harsh Nishant Lalai, Raj Sanjay Shah, Hanspeter Pfister, Sashank Varma, Grace Guo View a PDF of the paper titled When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations, by Harsh Nishant Lalai and 4 other authors View PDF Abstract:Visualizations help communicate data insights, but deceptive data representations can distort their interpretation and propagate misinformation. While recent Vision Language Models (VLMs) perform well on many chart understanding tasks, their ability to detect misleading visualizations, especially when deception arises from subtle reasoning errors in captions, remains poorly understood. Here, we evaluate VLMs on misleading visualization-caption pairs grounded in a fine-grained taxonomy of reasoning errors (e.g., Cherry-picking, Causal inference) and visualization design errors (e.g., Truncated axis, Dual axis, inappropriate encodings). To this end, we develop a benchmark that combines real-world visualization with human-authored, curated misleading captions designed to elicit specific reasoning and visualization error types, enabling controlled analysis across error categories and modalities of misleadingness. Evaluating many commercial and open-source VLMs, we find that models detect...