What is expert parallelism?
Expert parallelism (EP) is a model-parallelism strategy specific to Mixture of Experts models. Instead of sharding each layer’s weights the way tensor parallelism does, expert parallelism distributes the experts themselves across GPUs, so each GPU is responsible for a subset of the experts. When a token is routed to an expert, its data is sent to whichever GPU holds that expert.
What is wide expert parallelism?
“Wide EP” refers to spreading a model’s experts across a large number of GPUs so that each GPU holds only a few experts per layer. For example, DeepSeek-R1 has 256 experts per layer; at an expert-parallel size of 64, each GPU holds four experts per layer.
Wide EP is used mainly for the decode phase of inference, where keeping only a few experts per GPU reduces the per-token memory traffic on each GPU and improves throughput. NVIDIA has reported that moving from a small configuration (EP8) to a wide one can raise per-GPU throughput by up to 1.8x on rack-scale systems.
What does expert parallelism require?
- A fast interconnect. Routing tokens to their experts is an all-to-all exchange that happens on every layer, which is bandwidth-intensive. Wide EP delivers its best results on high-bandwidth fabrics such as NVLink or RDMA networking, which is why the best numbers come from rack-scale systems.
- Expert load balancing. Because token routing is uneven, some experts run hot. An expert load balancer redistributes experts across GPUs, either from precomputed mappings or dynamically at runtime, so no single GPU stalls the layer.
Resources
To learn more about expert parallelism and serving MoE models, you can explore the following resources:
