Mixture of Experts (MoEs) in Transformers

Back to Articles Mixture of Experts (MoEs) in Transformers Published February 26, 2026 Update on GitHub Upvote 23 +17 Aritra Roy Gosthipaty ariG23498 Follow Pedro Cuenca pcuenq Follow merve merve Follow Ilyas Moutawwakil IlyasMoutawwakil Follow Arthur Zucker ArthurZ Follow Sergio Paniego sergiopaniego Follow Pablo Montalvo Molbap Follow Introduction Over the past few years, scaling dense language models has driven most progress in LLMs. From early models like the original ULMFiT (~30M parameters) or GPT-2 (1.5B parameters, which at the time was considered "too dangerous to release" 🧌), and eventually to today’s hundred-billion–parameter systems, the recipe was simple: More data + more parameters gives better performance. Scaling laws reinforced this trend, but dense scaling has practical limits: Training becomes increasingly expensive. Inference latency grows. Deployment requires significant memory and hardware. This is where Mixture of Experts (MoEs) enter the picture. If you're already familiar with MoEs and want to jump straight into the engineering work done in transformers, you can head directly to Transformers and MoEs. From Dense to Sparse: What Are MoEs? A Mixture of Experts model keeps the Transformer backbone, but replaces certain dense feed-forward layers with a set of experts. An “expert” is not a topic-specialized module (e.g., "math expert", "code expert"). It is simply a learnable sub-network. For each token, a router selects a small subset of experts to pro...