What is the KV cache?
The KV cache (key-value cache) is a store of the attention keys and values a transformer computes for each token it processes. During text generation, a model attends over every previous token on every step. Without caching, it would recompute the keys and values for the entire prefix each time it generates a new token. The KV cache saves that work by storing each token’s keys and values once and reusing them.
Why does the KV cache matter?
Model weights are fixed in size, but the KV cache grows with sequence length and with the number of concurrent requests. It is stored per layer, and for large models a single long request can consume gigabytes. This makes the KV cache, not the model weights, the real limit on how many requests fit on a GPU.
Because of this, most inference optimizations are really KV-cache management techniques:
- Paged attention treats the cache like operating-system virtual memory, splitting it into fixed-size pages allocated on demand, which lets many requests share GPU memory without fragmentation.
- Prefix caching reuses the cache of a shared prompt (a system prompt or conversation history) across requests instead of recomputing it.
- KV-cache offloading moves cold cache blocks to cheaper memory tiers (host DRAM, SSD, or network storage) to free GPU memory for active requests.
KV cache in prefill and decode
The KV cache is what splits inference into two phases with different resource profiles. The prefill phase processes the whole prompt at once to fill the cache; it is compute-bound. The decode phase generates one token at a time, streaming the full cache through the GPU on every step; it is memory-bandwidth-bound. This asymmetry is the basis for disaggregated serving.
Resources
To learn more about the KV cache and LLM inference, you can explore the following resources:
