What is continuous batching?
Continuous batching (also called in-flight batching) is a scheduling technique for LLM inference. Instead of grouping requests into a fixed batch that starts and finishes together, the server adds new requests to the running batch as they arrive and removes finished ones as they complete. The batch is a moving set rather than a fixed one.
Why does it help?
Text generation is autoregressive: different requests finish after different numbers of tokens. With static batching, the whole batch has to wait for its slowest request before any GPU capacity frees up, and short requests are held hostage by long ones. Continuous batching avoids this. As soon as one request finishes, its slot is given to a waiting request, so the GPU stays busy and throughput rises.
Continuous batching depends on cheap, dynamic memory allocation, because sequences are constantly joining and leaving. PagedAttention is what makes it practical: paged KV cache can be allocated and freed per sequence without fragmenting GPU memory.
Why It Shows Up in Operator Economics
By keeping the GPU fed instead of idling between static batches, continuous batching sharply raises utilization and throughput under mixed, concurrent traffic. That is why engines like vLLM and TensorRT-LLM outperform naive batching, and it is one of the software levers behind the wide output-speed spread seen across providers running the identical model. Same weights, same class of GPU, very different tokens per second. Batching strategy is part of the reason, and it lands directly on cost per token.
Resources
To learn more about continuous batching, you can explore the following resources:
