[2603.24804] GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretraining

Computer Science > Computer Vision and Pattern Recognition arXiv:2603.24804 (cs) [Submitted on 25 Mar 2026] Title:GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretraining Authors:Deen Dayal Mohan, Hossein Souri, Vitali Petsiuk, Juhong Min, Gopal Sharma, Luowei Zhou, Suren Kumar View a PDF of the paper titled GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretraining, by Deen Dayal Mohan and 6 other authors View PDF HTML (experimental) Abstract:Until recently, the success of large-scale vision-language models (VLMs) has primarily relied on billion-sample datasets, posing a significant barrier to progress. Latest works have begun to close this gap by improving supervision quality, but each addresses only a subset of the weaknesses in contrastive pretraining. We present GoldiCLIP, a framework built on a Goldilocks principle of finding the right balance of supervision signals. Our multifaceted training framework synergistically combines three key innovations: (1) a text-conditioned self-distillation method to align both text-agnostic and text-conditioned features; (2) an encoder integrated decoder with Visual Question Answering (VQA) objective that enables the encoder to generalize beyond the caption-like queries; and (3) an uncertainty-based weighting mechanism that automatically balances all heterogeneous losses. Trained on just 30 million images, 300x less data than leading methods, GoldiC...