What is prefix caching?
Prefix caching is an inference optimization that avoids recomputing the KV cache for text that many requests share. When a long system prompt, a set of few-shot examples, a common retrieved document, or a growing conversation history appears in many requests, prefix caching computes that prefix’s keys and values once and reuses them, so only the new tokens have to be processed.
For workloads with heavy sharing (multi-turn chat, agentic loops, retrieval-augmented generation with a common context), this substantially reduces the compute spent on prefill and improves time-to-first-token.
How is it implemented?
Serving engines implement prefix caching in different ways. vLLM hashes fixed-size blocks of tokens and reuses cached blocks whose hashes match. SGLang’s RadixAttention stores prefixes in a radix tree, which reuses any shared path automatically without the caller having to declare it.
Prefix caching across replicas
Prefix caching only helps within a single replica: each engine’s cache is private. Once a model runs on more than one replica, a naive load balancer can route a follow-up request to a replica that does not hold the relevant prefix, forcing a full recomputation and collapsing the cache hit rate.
Solving this requires KV-aware (prefix-aware) routing, which sends each request to the replica that already holds its prefix. Frameworks such as NVIDIA Dynamo provide this routing across a pool of replicas, which is what makes prefix caching effective at scale.
Resources
To learn more about prefix caching, you can explore the following resources:
