What is Continuous Batching?
Continuous Batching is an LLM serving scheduling pattern that can admit, advance, pause, or retire requests at model-iteration boundaries instead of keeping one fixed request batch until every sequence finishes.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Static, Dynamic, and Continuous Batches
Static batching fixes a request group before execution and commonly holds capacity until the group completes. Request-level dynamic batching waits for a size or time threshold, then still executes the selected group as a batch. Continuous Batching revisits membership between model iterations. The [Orca paper](https://www.usenix.org/conference/osdi22/presentation/yu) introduced iteration-level scheduling and selective batching for generative Transformer serving; its reported speedup belongs to the paper's GPT-3 system and comparison, not every current engine.
Scheduler, Memory, and Fairness
A production scheduler tracks waiting and running requests, token budget, KV-block availability, priorities, deadlines and cancellation. It may prioritize Decode to control ITL, chunk long Prefill work to reduce interference, or preempt and recompute lower-priority sequences when cache capacity is exhausted. [SARATHI](https://arxiv.org/abs/2308.16369) studies Decode-maximal batches with Chunked Prefill, while current engines implement different policies. FCFS alone does not guarantee fairness across prompt lengths, tenants or priority classes; starvation and preemption loss need explicit limits.
Benchmark Contract
Evaluate Continuous Batching under a declared arrival process, burstiness, concurrency, prompt/output-length distribution, cache state, scheduler policy, token budget, memory limit, model, runtime and hardware. Report request and Token Throughput with p50/p95/p99 Queue Time, TTFT, ITL or TPOT, End-to-End Latency, cancellations, errors, preemption, starvation and SLO-qualified Goodput. Compare against static or request-level dynamic batching at the same workload and quality; a larger batch that only improves aggregate TPS while violating interactive SLOs is not a production win.
Key Characteristics
- Rebuilds the active work set at model-iteration boundaries
- Lets completed or cancelled requests release capacity before other sequences finish
- Coordinates admission with token budgets and KV cache allocation
- Can mix or alternate prefill, chunked prefill, decode, and multi-token steps
- Trades aggregate utilization against queueing, tail latency, fairness, and preemption
- Requires workload-level measurement rather than an isolated maximum batch size
Common Use Cases
- Serving concurrent chat requests with variable prompt and output lengths
- Keeping streaming decode active while admitting bounded prompt work
- Applying tenant, priority or deadline policies to shared inference capacity
- Responding to cancellation and releasing KV cache before request completion limits
- Finding sustainable load under TTFT, ITL and End-to-End SLOs
Example
Loading code...Frequently Asked Questions
How is Continuous Batching different from dynamic batching?
Request-level dynamic batching waits for a size or timeout and then runs the selected batch. Continuous Batching can reconsider membership after each model iteration, letting finished requests leave and new work enter. Some products use dynamic batching as an umbrella term, so confirm the scheduler boundary rather than relying on the label.
Does every request advance by one Token per iteration?
No. Standard Decode often advances one Token per active sequence, but speculative decoding can accept multiple Tokens and Chunked Prefill can consume many input positions. An iteration is the scheduler's execution boundary; the number and type of Tokens processed must be measured.
Does Continuous Batching always reduce latency?
No. It can remove fixed-batch head-of-line blocking and improve utilization, but aggressive admission or long Prefill work can increase Queue Time, TTFT and ITL. Scheduler policy, load, KV capacity, prompt shape and fairness controls determine the result.
What happens when the KV cache is full?
The engine can stop admitting work, wait for completions, reject requests, evict cache blocks, swap state or preempt and later recompute a sequence. Each option changes latency and useful capacity. Track preemption count, lost work, eviction, queue age and out-of-memory failures rather than only GPU utilization.
How should Continuous Batching be tuned?
Replay realistic arrivals and length distributions, then sweep token budget, maximum sequences, Prefill chunking and priority policy. Select the highest SLO-qualified Goodput that also satisfies fairness, cancellation and quality requirements; do not copy a batch setting from a different model, runtime or GPU.