![[2512.23447] Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2512.23447] Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
Summary
This paper introduces an auxiliary loss function, ERC loss, to improve the performance of Mixture-of-Experts (MoE) models by aligning router decisions with expert capabilities, enhancing model efficiency and specialization.
Why It Matters
The proposed ERC loss addresses a critical limitation in MoE models, which often struggle with effective routing. By ensuring that router decisions are closely tied to expert capabilities, this approach could lead to more efficient and specialized models, impacting various applications in machine learning and AI.
Key Takeaways
- ERC loss couples router decisions with expert capabilities, improving MoE model performance.
- The approach is computationally efficient, operating independently of batch size.
- It provides insights into expert specialization levels during training.
- The method has been validated through extensive analysis on large-scale datasets.
- ERC loss enhances the flexibility and control over expert routing in MoE architectures.
Computer Science > Computation and Language arXiv:2512.23447 (cs) [Submitted on 29 Dec 2025 (v1), last revised 24 Feb 2026 (this version, v2)] Title:Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss Authors:Ang Lv, Jin Ma, Yiyuan Ma, Siyuan Qiao View a PDF of the paper titled Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss, by Ang Lv and 3 other authors View PDF Abstract:Mixture-of-Experts (MoE) models lack explicit constraints to ensure the router's decisions align well with the experts' capabilities, which ultimately limits model performance. To address this, we propose expert-router coupling (ERC) loss, a lightweight auxiliary loss that tightly couples the router's decisions with expert capabilities. Our approach treats each expert's router embedding as a proxy token for the tokens assigned to that expert, and feeds perturbed router embeddings through the experts to obtain intermediate activations. The ERC loss enforces two constraints on these activations: (1) Each expert must exhibit higher activation for its own proxy token than for the proxy tokens of any other expert. (2) Each proxy token must elicit stronger activation from its corresponding expert than from any other expert. These constraints jointly ensure that each router embedding faithfully represents its corresponding expert's capability, while each expert specializes in processing the tokens actually routed to it. The ERC loss is computationally efficient, ope...
