[2508.09223] Hierarchical Adaptive networks with Task vectors for Test-Time Adaptation

Computer Science > Machine Learning arXiv:2508.09223 (cs) [Submitted on 11 Aug 2025 (v1), last revised 25 Mar 2026 (this version, v2)] Title:Hierarchical Adaptive networks with Task vectors for Test-Time Adaptation Authors:Sameer Ambekar, Marta Hasny, Laura Daza, Daniel M. Lang, Julia A. Schnabel View a PDF of the paper titled Hierarchical Adaptive networks with Task vectors for Test-Time Adaptation, by Sameer Ambekar and 4 other authors View PDF HTML (experimental) Abstract:Test-time adaptation allows pretrained models to adjust to incoming data streams, addressing distribution shifts between source and target domains. However, standard methods rely on single-dimensional linear classification layers, which often fail to handle diverse and complex shifts. We propose Hierarchical Adaptive Networks with Task Vectors (Hi-Vec), which leverages multiple layers of increasing size for dynamic test-time adaptation. By decomposing the encoder's representation space into such hierarchically organized layers, Hi-Vec, in a plug-and-play manner, allows existing methods to adapt to shifts of varying complexity. Our contributions are threefold: First, we propose dynamic layer selection for automatic identification of the optimal layer for adaptation to each test batch. Second, we propose a mechanism that merges weights from the dynamic layer to other layers, ensuring all layers receive target information. Third, we propose linear layer agreement that acts as a gating function, preventing...