What is Continuous Batching?
Continuous Batching is an LLM serving technique that dynamically groups active requests during inference, adding new requests and removing completed ones without waiting for a fixed batch to finish.
How It Works
Continuous batching improves GPU utilization for LLM serving by avoiding the rigid behavior of static batches. In a static batch, all requests wait for the slowest sequence; in continuous batching, the scheduler can insert new requests as others complete and allocate work token by token. This is especially important for chat workloads where prompts and output lengths vary widely. The tradeoff is scheduling complexity: the serving engine must manage KV cache memory, fairness, latency targets, and admission control while keeping the accelerator busy.
Key Characteristics
- Dynamically schedules requests instead of waiting for a fixed batch boundary
- Improves GPU utilization under variable prompt and output lengths
- Works closely with KV cache allocation and decode scheduling
- Can increase aggregate throughput while affecting per-request latency tradeoffs
- Common in modern LLM serving engines such as vLLM-style systems
Common Use Cases
- Serving many concurrent chat users with variable output lengths
- Improving GPU utilization for streaming LLM APIs
- Reducing idle time caused by short requests finishing early
- Balancing latency and throughput in production inference
- Benchmarking serving engines under realistic traffic mixes
Example
Loading code...Frequently Asked Questions
How is continuous batching different from static batching?
Static batching waits for a fixed group to finish, while continuous batching admits and removes requests as generation progresses.
Does continuous batching reduce latency?
It can reduce queueing and improve utilization, but the latency impact depends on scheduling policy, load, and fairness settings.
Why is continuous batching useful for chat?
Chat requests have uneven input and output lengths, so dynamic scheduling prevents short and long requests from wasting accelerator time.
What does continuous batching need from memory management?
It needs efficient KV cache allocation and eviction so active sequences can grow and finish without fragmenting memory badly.