What Is NVSentinel?
NVSentinel automates the response to GPU hardware and software faults in Kubernetes clusters. It watches for failure signals, classifies them, and then acts: cordoning the node so no new work lands on it, draining the pods already there, and handing the node off to a remediation path such as a reboot, a reset, or a break-fix ticket.
The problem it targets is time. On a large fleet, GPU failures are routine rather than exceptional, and the expensive part is not the failure itself but the window between a GPU going bad and someone noticing. In that window, jobs keep getting scheduled onto a broken node and keep failing. NVSentinel closes that window to seconds.
Architecture
NVSentinel is a set of loosely coupled microservices. Health monitors feed events in, and processing modules act on them. Modules communicate asynchronously through MongoDB change streams rather than calling each other directly, so a slow or failed module does not block the rest.
Health monitors:
- GPU health monitor, built on DCGM
- Syslog health monitor, analyzing journalctl output for system-level faults
- Cloud service provider health monitor, picking up scheduled maintenance and host events from provider APIs
- Kubernetes object monitor, evaluating CEL-based policies against cluster state
Processing modules: platform connectors (gRPC event ingest), the MongoDB event store, fault quarantine, node drainer, fault remediation, health events analyzer, labeler, metadata collector, and log collector.
What It Detects and How It Responds
Detected faults include GPU hardware failures (thermal excursions, ECC errors, XID events), system-level issues found through log patterns, cloud provider maintenance events, and resource health problems visible through Kubernetes objects.
The remediation sequence is:
- Cordon. Fault quarantine marks the node unschedulable.
- Drain. The node drainer evicts running pods with configurable timeouts.
- Remediate. Fault remediation triggers an external break-fix system through a CRD.
- Recover. The janitor reboots or terminates the node through cloud provider APIs.
NVSentinel, DCGM, and Fleet Intelligence
These three are often confused because they all touch GPU health. DCGM is the telemetry source, exposing metrics and health checks on a host. NVIDIA Fleet Intelligence is a managed service that aggregates that telemetry across a fleet for visibility and reporting. NVSentinel is the actor: it consumes fault signals and changes cluster state in response. Visibility and remediation are separate jobs, and NVSentinel does the second one.
NVSentinel is Apache 2.0 licensed and is the health automation component of NVIDIA DSX OS.
