What is RDMA?
RDMA (Remote Direct Memory Access) is a networking capability that lets one machine access another machine’s memory directly, without involving either machine’s CPU, operating-system kernel, or data copies along the way. This bypass is what gives RDMA its very low latency and high throughput compared to conventional TCP/IP networking.
Common RDMA-capable fabrics include InfiniBand and RoCE (RDMA over Converged Ethernet). NVIDIA’s NVLink provides a similar direct-access path between GPUs within a node or rack.
Why does RDMA matter for AI infrastructure?
Large AI workloads move enormous amounts of data between GPUs, and the network can easily become the bottleneck. RDMA removes the CPU and OS from that path, so GPUs on different nodes can exchange data at close to hardware speed. Two cases where this is decisive:
- Distributed training. Multi-node training with data or model parallelism requires frequent all-reduce and all-to-all exchanges of gradients and activations. RDMA (often via GPUDirect) keeps those collective operations fast enough that adding nodes actually adds usable throughput.
- Disaggregated inference. Splitting prefill and decode across separate GPU pools means moving the KV cache, often gigabytes per request, between GPUs. That transfer is only cheap enough to be worthwhile on an RDMA-capable fabric.
The practical takeaway
Whether an inference or training feature is worth enabling often depends on whether the underlying cluster has RDMA networking. Features that require moving large tensors between nodes, such as disaggregated serving or wide expert parallelism, reach their full performance only where the fabric supports RDMA.
Resources
To learn more about RDMA and high-performance GPU networking, you can explore the following resources:
