What Is the KAI Scheduler?
The KAI Scheduler is a Kubernetes scheduler built for GPU workloads rather than general-purpose services. It came out of Run:ai, which NVIDIA acquired, and was released under Apache 2.0. It descends from kube-batch and is now a CNCF Sandbox project.
The default Kubernetes scheduler places pods one at a time against available resources. That model breaks down for AI clusters in three specific ways: distributed training jobs need all their workers placed simultaneously or none at all, GPU capacity has to be divided among teams under a quota policy rather than first-come-first-served, and small inference workloads waste a whole GPU when they only need a fraction of one. The KAI Scheduler addresses all three.
Key Capabilities
| Capability | What it does |
|---|---|
| Gang scheduling | Places all pods in a group together or none of them, so distributed jobs do not deadlock holding partial allocations |
| Hierarchical queues | Nested queues with quota, priority, and limits, mapping to an organization’s team and project structure |
| Fair share and reclaim | Dominant Resource Fairness across queues, with reclaim of borrowed capacity when a queue’s guaranteed share is needed |
| Fractional GPU allocation | Multiple workloads share one physical GPU, for inference and interactive work that cannot saturate a full device |
| Bin packing or spread | Pack for low fragmentation, or spread for resilience and load balancing, chosen per policy |
| Topology-aware scheduling | Places tightly coupled pods within the same high-bandwidth interconnect domain |
| Dynamic Resource Allocation | Uses the Kubernetes DRA API for vendor-specific hardware requests |
| Elastic workloads | Jobs scale between a minimum and maximum replica count as capacity frees up |
| Priority and preemptibility | Workload priority is separate from whether a workload can be preempted |
The scheduler’s podgrouper handles common workload types automatically, so jobs submitted through familiar operators are grouped for gang scheduling without users writing PodGroup resources by hand.
Where It Sits in a Serving Stack
For inference, the KAI Scheduler consumes the PodGang API from NVIDIA Grove, which is how a disaggregated NVIDIA Dynamo deployment gets its prefill and decode pools placed with the right gang and topology constraints. It is the scheduling component of NVIDIA DSX OS.
Why Operators Care
For a multi-tenant GPU platform, the scheduler is where utilization economics are decided. Quotas determine whether one team can starve another. Fractional allocation determines whether a fleet of small inference endpoints burns whole GPUs. Reclaim policy determines whether idle guaranteed capacity can be lent out and taken back. Getting these wrong shows up directly as GPU hours billed against work that was not done.
