![[2412.06014] Post-hoc Probabilistic Vision-Language Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2412.06014] Post-hoc Probabilistic Vision-Language Models
Summary
This article presents a novel approach to uncertainty estimation in vision-language models (VLMs) by proposing a post-hoc method that enhances predictive accuracy without requiring additional training.
Why It Matters
As vision-language models are increasingly deployed in critical applications, understanding and quantifying uncertainty is vital for ensuring reliability and safety. This research offers a significant advancement in the field by providing a method that enhances model interpretability and performance in active learning scenarios.
Key Takeaways
- Introduces a post-hoc uncertainty estimation method for VLMs.
- Improves predictive uncertainties and interpretable estimates.
- Supports sample-efficient active learning strategies.
- No additional training is required for implementation.
- Promising implications for safety-critical applications.
Computer Science > Computer Vision and Pattern Recognition arXiv:2412.06014 (cs) [Submitted on 8 Dec 2024 (v1), last revised 13 Feb 2026 (this version, v5)] Title:Post-hoc Probabilistic Vision-Language Models Authors:Anton Baumann, Rui Li, Marcus Klasson, Santeri Mentu, Shyamgopal Karthik, Zeynep Akata, Arno Solin, Martin Trapp View a PDF of the paper titled Post-hoc Probabilistic Vision-Language Models, by Anton Baumann and 7 other authors View PDF Abstract:Vision-language models (VLMs), such as CLIP and SigLIP, have found remarkable success in classification, retrieval, and generative tasks. For this, VLMs deterministically map images and text descriptions to a joint latent space in which their similarity is assessed using the cosine similarity. However, a deterministic mapping of inputs fails to capture uncertainties over concepts arising from domain shifts when used in downstream tasks. In this work, we propose post-hoc uncertainty estimation in VLMs that does not require additional training. Our method leverages a Bayesian posterior approximation over the last layers in VLMs and analytically quantifies uncertainties over cosine similarities. We demonstrate its effectiveness for uncertainty quantification and support set selection in active learning. Compared to baselines, we obtain improved and well-calibrated predictive uncertainties, interpretable uncertainty estimates, and sample-efficient active learning. Our results show promise for safety-critical applications o...
