![[2602.17871] Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.17871] Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
Summary
This paper explores the fine-grained knowledge capabilities of vision-language models (VLMs), highlighting their performance on visual question answering and classification benchmarks.
Why It Matters
Understanding the limitations and strengths of VLMs in fine-grained visual tasks is crucial for advancing AI applications in areas like document understanding and multimodal dialogue. This research identifies key factors that can enhance model performance, guiding future developments in AI.
Key Takeaways
- VLMs show significant progress in visual question answering but lag in fine-grained classification tasks.
- A better language model improves overall benchmark scores, while a superior vision encoder specifically enhances fine-grained performance.
- Pretraining strategies, especially when language model weights are unfrozen, are critical for fine-grained knowledge capabilities.
Computer Science > Computer Vision and Pattern Recognition arXiv:2602.17871 (cs) [Submitted on 19 Feb 2026] Title:Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models Authors:Dhruba Ghosh, Yuhui Zhang, Ludwig Schmidt View a PDF of the paper titled Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models, by Dhruba Ghosh and 2 other authors View PDF HTML (experimental) Abstract:Vision-language models (VLMs) have made substantial progress across a wide range of visual question answering benchmarks, spanning visual reasoning, document understanding, and multimodal dialogue. These improvements are evident in a wide range of VLMs built on a variety of base models, alignment architectures, and training data. However, recent works show that these models trail behind in traditional image classification benchmarks, which test fine-grained visual knowledge. We test a large number of recent VLMs on fine-grained classification benchmarks and identify potential factors in the disconnect between fine-grained knowledge and other vision benchmarks. Through a series of ablation experiments, we find that using a better LLM improves all benchmark scores equally, while a better vision encoder disproportionately improves fine-grained classification performance. Furthermore, we find that the pretraining stage is also vital to fine-grained performance, particularly when the language model weights are unfrozen during pretraining. These insights pa...