[2603.04894] Differentially Private Multimodal In-Context Learning

Computer Science > Artificial Intelligence arXiv:2603.04894 (cs) [Submitted on 5 Mar 2026] Title:Differentially Private Multimodal In-Context Learning Authors:Ivoline C. Ngong, Zarreen Reza, Joseph P. Near View a PDF of the paper titled Differentially Private Multimodal In-Context Learning, by Ivoline C. Ngong and 2 other authors View PDF HTML (experimental) Abstract:Vision-language models are increasingly applied to sensitive domains such as medical imaging and personal photographs, yet existing differentially private methods for in-context learning are limited to few-shot, text-only settings because privacy cost scales with the number of tokens processed. We present Differentially Private Multimodal Task Vectors (DP-MTV), the first framework enabling many-shot multimodal in-context learning with formal $(\varepsilon, \delta)$-differential privacy by aggregating hundreds of demonstrations into compact task vectors in activation space. DP-MTV partitions private data into disjoint chunks, applies per-layer clipping to bound sensitivity, and adds calibrated noise to the aggregate, requiring only a single noise addition that enables unlimited inference queries. We evaluate on eight benchmarks across three VLM architectures, supporting deployment with or without auxiliary data. At $\varepsilon=1.0$, DP-MTV achieves 50% on VizWiz compared to 55% non-private and 35% zero-shot, preserving most of the gain from in-context learning under meaningful privacy constraints. Subjects: Ar...