![[2509.25684] LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2509.25684] LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts
Summary
The paper presents LD-MoLE, a novel Learnable Dynamic Routing mechanism for Mixture of LoRA Experts, enhancing token-dependent expert allocation in large language models.
Why It Matters
As large language models become increasingly complex, efficient fine-tuning methods are crucial for adapting them to specific tasks. LD-MoLE offers a significant advancement by enabling dynamic routing, which can optimize performance and resource utilization in AI applications.
Key Takeaways
- LD-MoLE replaces conventional TopK routing with a learnable, differentiable routing mechanism.
- The method allows for adaptive expert allocation based on token and layer characteristics.
- Extensive experiments show LD-MoLE outperforms existing state-of-the-art methods across various benchmarks.
- The approach includes a sparsity control objective to regularize the number of activated experts.
- This innovation can lead to more efficient and effective applications of large language models.
Computer Science > Computation and Language arXiv:2509.25684 (cs) [Submitted on 30 Sep 2025 (v1), last revised 23 Feb 2026 (this version, v2)] Title:LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts Authors:Yuan Zhuang, Yi Shen, Yuexin Bian, Qing Su, Shihao Ji, Yuanyuan Shi, Fei Miao View a PDF of the paper titled LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts, by Yuan Zhuang and 6 other authors View PDF HTML (experimental) Abstract:Recent studies have shown that combining parameter-efficient fine-tuning (PEFT) with mixture-of-experts (MoE) is an effective strategy for adapting large language models (LLMs) to the downstream tasks. However, most existing approaches rely on conventional TopK routing, which requires careful hyperparameter tuning and assigns a fixed number of experts to each token. In this work, we propose LD-MoLE, a Learnable Dynamic routing mechanism for Mixture of LoRA Experts that enables adaptive, token-dependent, and layer-wise expert allocation. Our method replaces the non-differentiable TopK selection with a differentiable routing function and a closed-form solution. Moreover, our design allows the model to adaptively determine the number of experts to activate for each token at different layers. In addition, we introduce an analytical sparsity control objective to regularize the number of activated experts. Extensive experiments on the Qwen3-1.7B and Llama-3.2-3B models show that LD-MoLE achieves the highest average scores...