What is disaggregated serving?
Disaggregated serving is an inference-serving pattern that separates the two phases of LLM text generation, prefill and decode, onto different GPUs rather than running both on the same worker.
- Prefill processes the entire input prompt at once to populate the KV cache. It is compute-bound and determines time-to-first-token.
- Decode generates the output one token at a time, streaming the whole KV cache through the GPU on each step. It is memory-bandwidth-bound and determines how fast tokens arrive after the first.
Because these phases have opposite resource profiles, running them together forces a compromise: a long prompt in the prefill stage can stall other users' token generation. Disaggregation gives each phase its own pool of GPUs, sized and scaled independently.
Why does it help?
Separating the phases lets an operator allocate the right amount and type of hardware to each, and scale them independently as traffic shifts. When prompts get longer, more GPUs move to prefill; when generations get longer, more move to decode. NVIDIA has reported up to 7x higher throughput from disaggregated serving compared with running both phases together on the same GPUs.
What does it require?
The catch is the KV-cache transfer. Once prefill finishes, the cache, often gigabytes for a long request, has to move to the decode pool. That transfer is only cheap enough to be worthwhile on a fast interconnect, so disaggregated serving pays off on clusters with RDMA-capable networking (InfiniBand, RoCE, or NVLink) and offers little benefit without it. Frameworks such as NVIDIA Dynamo implement disaggregated serving on top of engines like vLLM, SGLang, and TensorRT-LLM.
Resources
To learn more about disaggregated serving, you can explore the following resources:
