[2604.02338] LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning

Computer Science > Machine Learning arXiv:2604.02338 (cs) [Submitted on 1 Feb 2026] Title:LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning Authors:Md Kowsher, Haris Mansoor, Nusrat Jahan Prottasha, Ozlem Garibay, Victor Zhu, Zhengping Ji, Chen Chen View a PDF of the paper titled LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning, by Md Kowsher and 6 other authors View PDF Abstract:MoE-PEFT methods combine Mixture of Experts with parameter-efficient fine-tuning for multi-task adaptation, but require separate adapters per expert causing trainable parameters to scale linearly with expert count and limiting applicability to adapter-based architectures. We propose LiME (Lightweight Mixture of Experts), which achieves expert specialization through lightweight modulation rather than adapter replication. Instead of separate adapters, LiME uses a single shared PEFT module and modulates its output with lightweight expert vectors, reducing expert parameters while generalizing to any PEFT method. Notably, LiME introduces zero-parameter routing by leveraging existing frozen and adapted representations eliminating learned router parameters typically required per layer. Theoretically, we prove that (i) more experts preserve more task-relevant information and (ii) modulation approximates full expert-specific PEFT with bounded error. LiME further incorporates n-gram windowed routing and adaptive expert selection (Auto Top-K) b...