What is PagedAttention?
PagedAttention is a memory-management technique for the KV cache in LLM inference, introduced by the vLLM project. It borrows the idea of paging from operating-system virtual memory: instead of allocating one large contiguous block of GPU memory per request, it splits the KV cache into small fixed-size pages that are allocated on demand and can be shared between sequences.
What problem does it solve?
Before PagedAttention, serving systems typically reserved a single contiguous block of memory for each request, sized to the maximum possible sequence length. Most requests never reached that length, so a large fraction of GPU memory sat reserved but unused, a form of internal fragmentation. Because the KV cache is the main limit on how many requests fit on a GPU, this waste directly reduced how many requests a GPU could serve at once.
Paging removes that waste. Memory is allocated page by page as a sequence grows, and pages can be shared, so a GPU can hold many more concurrent requests. This is what makes techniques like continuous batching and prefix caching practical: continuous batching needs cheap, dynamic allocation as requests join and leave, and prefix caching relies on sequences being able to share the pages that hold a common prompt.
Resources
To learn more about PagedAttention and efficient LLM serving, you can explore the following resources:
