Mixture of Experts (MoE)
Architecture where the model contains many specialized sub-networks (experts) and a router activates only a few per input, scaling capacity without scaling compute.
Last updated: April 26, 2026
Definition
Mixture of Experts is the architectural trick behind most frontier models in 2026. Instead of one huge dense network where every parameter activates on every input, MoE models have many smaller expert sub-networks and a router that picks 2 to 8 experts to activate per input token. The result: a model with hundreds of billions of total parameters that activates only tens of billions per inference, giving the capacity of a giant model at the inference cost of a much smaller one. Mixtral 8x7B, GPT-4, GPT-5, Claude Sonnet, and DeepSeek V3 all use MoE.
When To Use
You do not pick MoE; the model provider does. Knowing the term helps when reading model architecture papers, comparing inference cost vs total parameter count, or evaluating self-hosted MoE models like Mixtral.
Related Terms
Building with Mixture of Experts (MoE)?
I've shipped this pattern in real production systems. If you want a second pair of eyes on your architecture, that's what I do.