What is continuous batching?
Continuous batching (also called in-flight batching) is a scheduling technique for LLM inference. Instead of grouping requests into a fixed batch that starts and finishes together, the server adds new requests to the running batch as they arrive and removes finished ones as they complete. The batch is a moving set rather than a fixed one.
Why does it help?
Text generation is autoregressive: different requests finish after different numbers of tokens. With static batching, the whole batch has to wait for its slowest request before any GPU capacity frees up, and short requests are held hostage by long ones. Continuous batching avoids this. As soon as one request finishes, its slot is given to a waiting request, so the GPU stays busy and throughput rises.
Continuous batching depends on cheap, dynamic memory allocation, because sequences are constantly joining and leaving. PagedAttention is what makes it practical: paged KV cache can be allocated and freed per sequence without fragmenting GPU memory.
Resources
To learn more about continuous batching, you can explore the following resources:
