![[2411.18207] From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2411.18207] From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects
Summary
This paper introduces a framework for open vocabulary object detection that allows vision language models to identify and learn novel objects in real-time, enhancing their application in critical scenarios like autonomous driving.
Why It Matters
As traditional object detection methods are limited to predefined classes, this research addresses significant gaps in detecting novel objects, which is crucial for applications in safety-critical environments. The proposed methods improve the adaptability and robustness of AI systems in dynamic real-world settings.
Key Takeaways
- The framework enables vision language models to detect previously unseen objects.
- Introduces Open World Embedding Learning (OWEL) for better semantic understanding.
- Implements Multi-Scale Contrastive Anchor Learning (MSCAL) to enhance classification accuracy.
- Achieves state-of-the-art results in open world object detection benchmarks.
- Addresses limitations of existing open vocabulary detection methods.
Computer Science > Computer Vision and Pattern Recognition arXiv:2411.18207 (cs) [Submitted on 27 Nov 2024 (v1), last revised 26 Feb 2026 (this version, v4)] Title:From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects Authors:Zizhao Li, Zhengkang Xiang, Joseph West, Kourosh Khoshelham View a PDF of the paper titled From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects, by Zizhao Li and 3 other authors View PDF HTML (experimental) Abstract:Traditional object detection methods operate under the closed-set assumption, where models can only detect a fixed number of objects predefined in the training set. Recent works on open vocabulary object detection (OVD) enable the detection of objects defined by an in-principle unbounded vocabulary, which reduces the cost of training models for specific tasks. However, OVD heavily relies on accurate prompts provided by an ``oracle'', which limits their use in critical applications such as driving scene perception. OVD models tend to misclassify near-out-of-distribution (NOOD) objects that have similar features to known classes, and ignore far-out-of-distribution (FOOD) objects. To address these limitations, we propose a framework that enables OVD models to operate in open world settings, by identifying and incrementally learning previously unseen objects. To detect FOOD objects, we propose Open World Embedding Learning (OWEL) and introduce the concept of Pseudo Un...
