What is NVIDIA Dynamo?
NVIDIA Dynamo is an open-source, distributed inference framework for serving large language models across multiple nodes. It is not itself an inference engine. It sits above one, coordinating a serving engine (vLLM, SGLang, or TensorRT-LLM) into a system that can route requests intelligently, split a request’s work across separate GPU pools, and scale those pools to meet latency targets.
As NVIDIA’s documentation puts it, Dynamo “doesn’t replace SGLang, TensorRT-LLM, or vLLM, it turns them into a coordinated multi-node inference system.” It is positioned as the successor to Triton for distributed generative-AI serving.
What problems does Dynamo solve?
A single serving engine optimizes one replica of one model. The moment you run more than one replica, or a model too large for one node, the engine cannot coordinate across those boundaries. Dynamo fills that gap with a few distinct capabilities:
- KV-aware routing. Requests are routed to the replica that already holds their prefix in its KV cache, instead of round-robin. This avoids recomputing shared prompts and improves time-to-first-token on workloads with reusable structure (multi-turn chat, shared system prompts, agentic loops).
- Disaggregated serving. The prefill and decode phases of inference have opposite hardware profiles: prefill is compute-bound, decode is memory-bandwidth-bound. Dynamo runs them on separate, independently scaled GPU pools and moves the KV cache between them, rather than letting one phase starve the other.
- SLA-driven autoscaling. Its Planner component scales the prefill and decode pools independently to hit latency targets (time-to-first-token and inter-token latency) at minimum GPU cost.
Dynamo’s main components
| Component | Role |
|---|---|
| Smart Router | KV-cache-aware request routing across replicas |
| Planner | Autoscales prefill and decode pools against latency SLOs |
| KV Cache Manager | Offloads cold KV cache to DRAM, SSD, or network storage |
| NIXL | Moves KV cache between GPUs over NVLink, InfiniBand, or RoCE |
When is Dynamo worth adopting?
Dynamo’s two features have different requirements. KV-aware routing works on any cluster and pays off wherever traffic has reusable structure. Disaggregated serving is only worth turning on where the network fabric supports RDMA, because moving gigabytes of KV cache per request between GPUs is only cheap on a fast interconnect. Large Mixture of Experts models benefit most, using wide expert parallelism across many GPUs, but that too depends on high-bandwidth interconnect.
Resources
To learn more about NVIDIA Dynamo and distributed LLM inference, you can explore the following resources:
