NVIDIA Dynamo coordinates vLLM, SGLang, and TensorRT-LLM into a multi-node system. What it actually does, how you configure it for dense, MoE, and multimodal models, and where the payoff is gated on your hardware.
This is Part 2. Part 1 walked the inference stack from the GPU up and split it into three planes: a data plane that moves tokens, a control plane that decides what runs where, and a tenant plane that meters and bills. It said advanced stacks run prefill and decode on separate GPU pools and route requests across replicas, then deferred the details. This post is those details. Dynamo is the component that does that work.
TL;DR
- Dynamo is not an inference engine. It is a coordination layer that turns vLLM, SGLang, or TensorRT-LLM into a multi-node system. It sits in the control plane and reaches down into the data plane.
- It has two main features: KV-aware routing (send a request to the replica that already holds its prefix) and disaggregated serving (run prefill and decode on separate GPU pools). The first works on any hardware. The second needs a fast interconnect (RDMA) to move the KV cache, so it only pays off on the right infrastructure.
- Model type changes what you configure. Dense models split on tensor parallelism. MoE models add expert parallelism and want a “wide EP” decode layout. Multimodal models add a third phase, encode, in front of prefill.
- Adopting Dynamo does not change Part 1’s conclusion. It is still the commodity data/control plane. The tenant plane is still yours. Dynamo makes the commodity part faster at scale; it does not move you up the stack.
What Dynamo is, precisely
One line in Dynamo’s own README captures it: Dynamo “doesn’t replace SGLang, TensorRT-LLM, or vLLM, it turns them into a coordinated multi-node inference system.” That is the whole idea. You still run a real engine underneath. Dynamo is the layer above it that handles the things a single engine cannot see: routing across replicas, splitting a request’s phases across GPU pools, moving KV cache between them, and scaling the pools to hit a latency target.
In the three-plane model from Part 1, Dynamo straddles the data and control planes. It makes control-plane decisions (which replica, which pool, how many GPUs) and it participates in the data plane (it moves KV cache between GPUs). It does not touch the tenant plane at all. It has no concept of your customers, your billing, or your quotas, which is exactly why Part 1’s conclusion survives it.
Dynamo has four pieces worth naming, because they map onto problems Part 1 already described:
| Component | Problem it solves | Part 1 reference |
|---|---|---|
| Smart Router | Routes to the replica holding the request’s prefix, not round-robin | “a dumb load balancer hurts” |
| Planner | Autoscales prefill and decode pools against latency SLOs | “autoscaling on the right signal” |
| KV Cache Manager | Offloads cold KV cache to DRAM, SSD, or network storage | “the KV cache is the real capacity limit” |
| NIXL | Moves KV cache between GPUs over NVLink, InfiniBand, or RoCE | (new: the transport disaggregation needs) |
The first three are answers to problems Part 1 raised and left open. The fourth is the piece that makes disaggregation physically possible, and it is where hardware starts to matter.
Feature one: KV-aware routing
Part 1 described the failure mode exactly. Four replicas behind round-robin, a multi-turn conversation bounces between them, each new replica is cold and recomputes the whole history, and prefix-cache hit rate collapses. The fix is a router that knows where each prefix already lives.
Dynamo’s Smart Router does this by hashing incoming requests into a radix tree that tracks which worker holds which KV blocks, computing an overlap score between a new request and the cache already resident on each worker, and routing to the best combination of cache hit and current load. NVIDIA cites roughly 2x faster time-to-first-token from this alone on workloads with reusable prefixes, with the same benchmark echoed by Baseten.
Two operational notes that matter for a real deployment:
- This works on any hardware. KV-aware routing is a scheduling decision, not a data movement. It needs no special interconnect. If your traffic has structure (multi-turn chat, shared system prompts, agentic loops, many fine-tunes off one base), this is the part of Dynamo to turn on first.
- It does not require NATS. Dynamo can do prediction-based routing with no external event bus, or you can enable KV events for event-backed cache-state tracking when you want it. So a minimal deployment stays small.
If your traffic is short, unique, single-turn completions, there is little prefix to reuse and this earns little, which is the same caveat Part 1 closed on. Routing intelligence only pays off when the traffic has structure to exploit.
Feature two: disaggregated prefill and decode
This is the feature Part 1 pointed at and deferred. Recall the asymmetry: prefill processes the whole prompt at once and is compute-bound; decode generates one token at a time and is memory-bandwidth-bound. Run both on the same GPUs and one phase always starves the other. A long prompt from one user stalls everyone else’s token stream. This split of the phases across pools is what disaggregated serving means.
Disaggregation splits them into separate GPU pools, sizes each independently, and moves the KV cache from the prefill pool to the decode pool once the prompt is processed. NVIDIA quotes up to 7x throughput over monolithic serving on the same hardware. The Planner rebalances the two pools as traffic shifts: a burst of long prompts moves GPUs toward prefill, a burst of long generations moves them toward decode.
The catch is the KV cache transfer. Moving gigabytes of KV cache from one GPU’s memory to another’s, per request, only makes sense if the interconnect is fast. Dynamo uses NIXL to do this transfer over NVLink, InfiniBand, or RoCE, asynchronously and without blocking the GPU’s forward pass. On infrastructure with RDMA-capable networking, this is cheap enough to be worth it. On infrastructure without it, the transfer cost can eat the benefit.
This is the hardware gate. KV-aware routing works everywhere. Disaggregated serving is worth turning on where the fabric supports RDMA. That is a deployment decision, not a config default, and it is worth settling before you adopt Dynamo: you get one of its two features on any cluster, and the second only where the network supports it.
Configuring Dynamo by model type
Dynamo passes model-level parallelism settings through to the engine underneath. What changes by model type is not Dynamo itself, it is the worker topology you ask it to coordinate.
Dense models
The straightforward case. You split the model with tensor parallelism, sized to fit weights plus KV cache in memory, and then scale out with more replicas rather than wider tensor parallelism, because decode wants many concurrent sequences. Prefill workers can run higher tensor parallelism to push time-to-first-token down; decode workers scale on replica count. There is no expert parallelism to think about.
MoE models: wide expert parallelism
Mixture-of-Experts models (DeepSeek-R1, Mixtral, Llama-4) split their feed-forward layers into many experts, of which only a few fire per token. DeepSeek-R1 has 256 experts per MoE layer. This adds a parallelism axis, expert parallelism (EP): distribute the experts across GPUs instead of, or alongside, sharding each expert with tensor parallelism.
The interesting part is that prefill and decode want different layouts:
- Decode wants wide EP. Spread the experts thin so each GPU holds only a few and its per-token memory traffic stays small. NVIDIA’s concrete example: DeepSeek-R1 on a GB200 NVL72 rack at EP=64 puts four experts per GPU per layer, and moving from EP8 to a wide configuration delivers up to 1.8x higher per-GPU throughput.
- Attention often runs data-parallel, not tensor-parallel. For DeepSeek-style models Dynamo runs data-parallel attention alongside wide EP. The current vLLM example runs 16 GPUs across 2 nodes with a data-parallel size of 16, and each attention rank is its own Dynamo component emitting its own KV events and metrics.
Wide EP has a second requirement: expert load balancing. MoE routing is skewed, some experts run hot, and one overloaded GPU stalls the all-to-all exchange for the whole layer. Dynamo and TensorRT-LLM offer a load balancer with a static mode (precomputed expert-to-GPU mapping from historical traffic) and an online mode (redistribute experts at runtime). Without it, wide EP underdelivers.
Wide EP is also the case most sensitive to interconnect. The per-layer all-to-all exchange between experts is bandwidth-hungry, which is why NVIDIA’s best MoE numbers come from rack-scale NVLink. On single-node or slower-fabric setups you run smaller EP within a node and get less out of wide EP.
Multimodal models: a third phase
Vision-language models (Llama-4-Maverick, Qwen-VL) add a vision encoder in front of the language model. Dynamo can disaggregate that too, turning the pipeline from prefill-decode into encode-prefill-decode.
The encode worker is configured unlike any LLM worker: it runs only the vision encoder and projector, so it needs no KV cache configuration at all, just a model path and a batch size. It encodes images, ships the embeddings over NIXL, and the decode and prefill workers take it from there. You scale the encode pool independently, because a text-heavy workload needs few encoders and an image-heavy one needs many. This is the biggest structural departure from text-only serving: a third pool, with a completely different resource profile.
One case where disaggregation does not apply
Embedding models and rerankers have no decode phase. There is no token generation, no KV cache to hand off, and nothing to disaggregate. Run them aggregated and scale replicas. If your stack serves these alongside a generative model (a RAG setup, for instance), do not reach for the same disaggregated topology. It buys nothing here.
Here is the whole picture in one table:
| Model type | Parallelism axis | Disaggregation | What is different |
|---|---|---|---|
| Dense | Tensor parallel | Prefill / decode | Baseline case |
| MoE | Expert parallel (wide EP) | Prefill / decode | Wide-EP decode layout, expert load balancing |
| Multimodal | TP or EP for the LLM | Encode / prefill / decode | A third pool, the encoder, with no KV cache |
| Embedding / reranker | Replicas | None | No decode phase, nothing to disaggregate |
How you actually deploy this
The mechanics matter, because they shape how you operate Dynamo, and a couple of things surprise people coming from a single-engine setup.
The unit of deployment is a graph, expressed as a Kubernetes custom resource called a DynamoGraphDeployment. You write one YAML, apply it, and the Dynamo Operator (a controller you install once via Helm) reconciles it into running pods: the Frontend, the Router, and the worker pods, wired together with service discovery, GPU requests, and multi-node placement. You do not hand-create the individual pods.
The important structural fact: a graph serves one model. Its services map holds one Frontend and one model’s workers. Configuration lives in two layers stacked in that map. The Dynamo layer is which components exist, their replica counts, GPU counts, and node counts. The engine layer is the argument list on each worker, which is passed straight through to vLLM, SGLang, or TensorRT-LLM. Your dense-versus-MoE choice is engine args (--enable-expert-parallel, tensor-parallel size, data-parallel size); disaggregation is structural (a separate prefill worker entry and decode worker entry, rather than one combined worker).
This leads to a mental-model correction worth stating plainly, because it is the most common misconception. There is no central Dynamo server you register models against. Adding a different model means applying a new DynamoGraphDeployment, not editing an existing one. Each model is its own graph with its own Frontend. If you want one endpoint that routes across several models by name, that router lives above Dynamo (your gateway or ingress), dispatching to the right graph’s Frontend. The one exception is multi-LoRA: a single graph loads one base model and serves many adapters off it by name, because they share the base weights. One graph equals one base model.
The graph is not self-contained, either. It references cluster prerequisites you set up once, not per model: the Operator itself, the secrets it names (a HuggingFace token to pull weights, an image pull secret), the coordination backends (etcd, and NATS if you want event-backed KV routing rather than the prediction-based mode), and node pools with the right GPUs and, for the disaggregated and wide-EP paths, RDMA networking. The CR requests GPUs; it does not create nodes or provision fabric.
Scaling: pick a mode, do not mix hand-tuning with autoscaling
Dynamo can autoscale, but “does it autoscale” has three answers, and choosing one is a real decision.
- Static. You set a fixed replica count per worker. Predictable, you own the number, nothing changes it under you. This is the baseline, and it is the only mode where “add capacity” is something you do by hand.
- Load-based. Scale replicas on real-time engine metrics. The signal that matters is KV-cache utilization and queue depth, not CPU, which is the same point Part 1 made about autoscaling on the right signal.
- SLA-based (the Planner). You give it a latency target (TTFT and ITL) rather than a replica count, and it scales the prefill and decode pools independently to hit that target at minimum GPU cost. It forecasts load (ARIMA or Prophet predictors) and uses profiling data to size the pools proactively rather than reacting after the queue backs up. Scaling prefill and decode separately is the thing a plain Kubernetes HPA cannot do, because it knows nothing about the two phases' different resource profiles.
The trap: in the two autoscaled modes, you do not tweak replica counts yourself. The Planner or the autoscaler owns that number, and hand-setting it fights the controller. What you tune instead is the SLO, or the minimum and maximum replica bounds that define the envelope the Planner moves within. That floor and ceiling still matter: the floor keeps warm capacity so you never cold-start from zero, the ceiling caps cost. (Dynamo can even run a hybrid where a throughput-based pass sets the replica floor and a faster load-based pass adjusts above it.) So autoscaling is not “the Planner takes over and you set nothing.” You define the envelope; it moves within it.
Where this leaves the stack
Dynamo answers three of the open questions Part 1 raised: the smart router replaces the dumb load balancer, the Planner autoscales on latency SLOs instead of CPU, and disaggregation formalizes the prefill/decode split. It also adds one new requirement Part 1 did not: a fast interconnect to move KV cache, if you want the disaggregation half.
What it does not do is change where the engineering value lives. Dynamo is a data-and-control-plane component. It makes the commodity part of the stack faster and more efficient at scale, which is worth doing, but it does not know what a tenant is, does not meter tokens, and does not bill. The tenant plane is still yours, still the part no open-source component ships knowing, and still where a serving product is actually differentiated.
Adopt Dynamo where your model sizes and your interconnect justify it: KV-aware routing on any cluster with structured traffic, disaggregated serving where the fabric is RDMA-capable, wide EP for large MoE models on fast interconnect. Then spend your own engineering above it, on the tenant plane, which is the part that depends on your customers and that no component ships knowing.
Sources: NVIDIA Dynamo (developer.nvidia.com), Dynamo GitHub README, Scaling Large MoE Models with Wide Expert Parallelism on NVL72 (NVIDIA), How GB200 NVL72 and Dynamo Boost MoE Inference (NVIDIA), SLA-based Planner (Dynamo docs), Dynamo documentation.
