[2603.12055] Continual Learning with Vision-Language Models via Semantic-Geometry Preservation

Computer Science > Computer Vision and Pattern Recognition arXiv:2603.12055 (cs) [Submitted on 12 Mar 2026 (v1), last revised 21 Mar 2026 (this version, v2)] Title:Continual Learning with Vision-Language Models via Semantic-Geometry Preservation Authors:Chiyuan He, Zihuan Qiu, Fanman Meng, Runtong Zhang, Linfeng Xu, Qingbo Wu, Hongliang Li View a PDF of the paper titled Continual Learning with Vision-Language Models via Semantic-Geometry Preservation, by Chiyuan He and 6 other authors View PDF HTML (experimental) Abstract:Continual learning of pretrained vision-language models (VLMs) is prone to catastrophic forgetting, yet current approaches adapt to new tasks without explicitly preserving the cross-modal semantic geometry inherited from pretraining and previous stages, allowing new-task supervision to induce geometric distortion. We observe that the most pronounced drift tends to concentrate in vulnerable neighborhoods near the old-new semantic interface, where shared visual patterns are easily re-explained by new textual semantics. To address this under an exemplar-free constraint, we propose Semantic Geometry Preservation for Continual Learning (SeGP-CL). SeGP-CL first probes the drift-prone region by constructing a compact set of adversarial anchors with dual-targeted projected gradient descent (DPGD), which drives selected new-task seeds toward old-class semantics while remaining faithful in raw visual space. During training, we preserve cross-modal structure by anch...