[2510.18034] Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning

Computer Science > Computer Vision and Pattern Recognition arXiv:2510.18034 (cs) [Submitted on 20 Oct 2025 (v1), last revised 8 Apr 2026 (this version, v2)] Title:Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning Authors:Roberto Brusnicki, David Pop, Yuan Gao, Mattia Piccinini, Johannes Betz View a PDF of the paper titled Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning, by Roberto Brusnicki and 4 other authors View PDF HTML (experimental) Abstract:Autonomous driving systems remain critically vulnerable to the long-tail of rare, out-of-distribution semantic anomalies. While VLMs have emerged as promising tools for perception, their application in anomaly detection remains largely restricted to prompting proprietary models - limiting reliability, reproducibility, and deployment feasibility. To address this gap, we introduce SAVANT (Semantic Anomaly Verification/Analysis Toolkit), a novel model-agnostic reasoning framework that reformulates anomaly detection as a layered semantic consistency verification. By applying SAVANT's two-phase pipeline - structured scene description extraction and multi-modal evaluation - existing VLMs achieve significantly higher scores in detecting anomalous driving scenarios from input images. Our approach replaces ad hoc prompting with semantic-aware reasoning, transforming VLM-based detection into a principled decomposition across four semantic domains. We show that across a balance...