[2603.03535] Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts

Computer Science > Machine Learning arXiv:2603.03535 (cs) [Submitted on 3 Mar 2026] Title:Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts Authors:Sanae Lotfi, Lucas Caccia, Alessandro Sordoni, Jordan T. Ash, Miroslav Dudik View a PDF of the paper titled Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts, by Sanae Lotfi and 4 other authors View PDF HTML (experimental) Abstract:While large language models (LLMs) fine-tuned with lightweight adapters achieve strong performance across diverse tasks, their performance on individual tasks depends on the fine-tuning strategy. Fusing independently trained models with different strengths has shown promise for multi-task learning through three main strategies: ensembling, which combines outputs from independent models; merging, which fuses model weights via parameter averaging; and routing, which integrates models in an input-dependent fashion. However, many design decisions in these approaches remain understudied, and the relative benefits of more sophisticated ensembling, merging and routing techniques are not fully understood. We empirically evaluate their trade-offs, addressing two key questions: What are the advantages of going beyond uniform ensembling or merging? And does the flexibility of routing justify its complexity? Our findings indicate that non-uniform ensembling and merging improve performance, but routing offers even greater gains. To mitigate the computati...