What Is NVIDIA Grove?
Grove is a Kubernetes orchestration layer for AI inference workloads that span more than one pod. A Kubernetes Deployment assumes replicas are interchangeable and independent. A modern inference deployment is neither: a single logical model instance may consist of a router, a set of prefill workers, and a set of decode workers, each with different hardware requirements, different scaling behavior, and a required startup order.
Grove provides one declarative resource that covers the whole range, from a single-pod deployment to a multi-node disaggregated system, and hands the scheduler enough information to place the pieces correctly.
The Custom Resources
| Resource | Purpose |
|---|---|
| PodClique | A group of pods with one role (leader, worker, frontend), with its own config and scaling logic |
| PodCliqueScalingGroup | Several PodCliques that scale and schedule together as one unit |
| PodCliqueSet | The top-level resource: a managed set of components with autoscaling and topology-aware replica placement |
| PodGang | The scheduler-facing API that declares minimum replica guarantees for gang scheduling |
What It Solves
Hierarchical gang scheduling. A multi-node model instance is scheduled all-or-nothing. Without this, partial placements sit holding GPUs while waiting for peers that may never arrive, which wastes capacity and can deadlock a cluster.
Multi-level autoscaling. Scaling happens at the level that matters. A decode pool can grow on its own, or an entire PodCliqueSet replica can be added as a unit, rather than forcing one scaling granularity on every component.
Startup ordering. Components start in a declared sequence. This matters for MPI-style workloads and for disaggregated pipelines where a worker cannot usefully start before the component it registers with.
Topology awareness. Pods that exchange large volumes of data are placed inside the same network-optimized domain, such as an NVLink domain, instead of being scattered across a cluster where the interconnect becomes the bottleneck.
Grove, Dynamo, and the KAI Scheduler
The three fit together at different layers. NVIDIA Dynamo is the inference framework: it routes requests, manages KV cache, and splits prefill from decode. Grove expresses the resulting topology as Kubernetes resources. The KAI Scheduler consumes Grove’s PodGang API and does the actual placement, honoring the gang constraints and topology hints.
Grove is developed in the same GitHub organization as Dynamo and is licensed Apache 2.0. It is one of the inference components of NVIDIA DSX OS.
