[2603.07926] IMSE: Intrinsic Mixture of Spectral Experts Fine-tuning for Test-Time Adaptation

Computer Science > Computer Vision and Pattern Recognition arXiv:2603.07926 (cs) [Submitted on 9 Mar 2026 (v1), last revised 14 Apr 2026 (this version, v3)] Title:IMSE: Intrinsic Mixture of Spectral Experts Fine-tuning for Test-Time Adaptation Authors:Sunghyun Baek, Jaemyung Yu, Seunghee Koh, Minsu Kim, Hyeonseong Jeon, Junmo Kim View a PDF of the paper titled IMSE: Intrinsic Mixture of Spectral Experts Fine-tuning for Test-Time Adaptation, by Sunghyun Baek and 5 other authors View PDF HTML (experimental) Abstract:Test-time adaptation (TTA) has been widely explored to prevent performance degradation when test data differ from the training distribution. However, fully leveraging the rich representations of large pretrained models with minimal parameter updates remains underexplored. In this paper, we propose Intrinsic Mixture of Spectral Experts (IMSE) that leverages the spectral experts inherently embedded in Vision Transformers. We decompose each linear layer via singular value decomposition (SVD) and adapt only the singular values, while keeping the singular vectors fixed. We further identify a key limitation of entropy minimization in TTA: it often induces feature collapse, causing the model to rely on domain-specific features rather than class-discriminative features. To address this, we propose a diversity maximization loss based on expert-input alignment, which encourages diverse utilization of spectral experts during adaptation. In the continual test-time adaptation...