What is Mixture of Experts?
Mixture of Experts (MoE) is a model architecture in which the feed-forward layers are divided into many parallel sub-networks, called experts, and a lightweight router sends each token to only a small number of them (often the top one or two out of dozens or hundreds). Because only a few experts process any given token, the model activates a small fraction of its total parameters per token.
This gives MoE models a favorable trade-off: a very large total parameter count (which tends to improve quality) at the compute cost of a much smaller model (because most experts sit idle for any given token). DeepSeek-R1, Mixtral, and Llama-4 are well-known MoE models. DeepSeek-R1, for example, has 256 experts per MoE layer.
How does serving an MoE model differ?
Serving an MoE model introduces a parallelism axis that dense models do not have: expert parallelism, which distributes the experts across GPUs. This creates challenges that do not exist for a dense model of the same activated size:
- Expert routing is an all-to-all exchange. Each layer shuffles tokens to the GPUs holding their assigned experts, which is bandwidth-intensive and benefits from a fast interconnect such as NVLink or RDMA.
- Expert load can be skewed. Some experts run “hot” and receive far more tokens than others, and one overloaded GPU can stall the whole layer. Production serving uses an expert load balancer to redistribute experts.
- Prefill and decode want different layouts. The decode phase typically favors “wide” expert parallelism, spreading experts thin across many GPUs so each holds only a few.
Resources
To learn more about Mixture of Experts models, you can explore the following resources:
