[2603.21135] One Pool Is Not Enough: Multi-Cluster Memory for Practical Test-Time Adaptation

Computer Science > Computer Vision and Pattern Recognition arXiv:2603.21135 (cs) [Submitted on 22 Mar 2026] Title:One Pool Is Not Enough: Multi-Cluster Memory for Practical Test-Time Adaptation Authors:Yu-Wen Tseng, Xingyi Zheng, Ya-Chen Wu, I-Bin Liao, Yung-Hui Li, Hong-Han Shuai, Wen-Huang Cheng View a PDF of the paper titled One Pool Is Not Enough: Multi-Cluster Memory for Practical Test-Time Adaptation, by Yu-Wen Tseng and 6 other authors View PDF HTML (experimental) Abstract:Test-time adaptation (TTA) adapts pre-trained models to distribution shifts at inference using only unlabeled test data. Under the Practical TTA (PTTA) setting, where test streams are temporally correlated and non-i.i.d., memory has become an indispensable component for stable adaptation, yet existing methods universally store amples in a single unstructured pool. We show that this single-cluster design is fundamentally mismatched to PTTA: a stream clusterability analysis reveals that test streams are inherently multi-modal, with the optimal number of mixture components consistently far exceeding one. To close this structural gap, we propose Multi-Cluster Memory (MCM), a plug-and-play framework that organizes stored samples into multiple clusters using lightweight pixel-level statistical descriptors. MCM introduces three complementary mechanisms: descriptor-based cluster assignment to capture distinct distributional modes, Adjacent Cluster Consolidation (ACC) to bound memory usage by merging the m...