Disaggregated LLM Serving: Prefill and Decode

TL;DR

Disaggregated serving splits LLM prefill and decode into independently scalable pools. It can isolate time to first token from inter-token latency pressure and fit each phase to different hardware or parallelism. The trade-off is a new critical path: KV-cache transfer, phase-aware routing, compatibility, and recovery. It is an SLO architecture, not a guaranteed throughput optimization.

Why prefill and decode behave differently

Autoregressive inference has two major phases:

In an aggregated engine, one worker handles both. A burst of long prompts can consume compute and delay active generations. In a disaggregated architecture, dedicated prefill workers absorb prompt processing while decode workers continue streaming.

The KV transfer is the critical path

Decode cannot continue without the attention state produced by prefill. Depending on the serving backend, the prefill worker may copy KV blocks to the decode worker, expose remote memory, or coordinate a backend-specific transfer protocol.

The transfer cost grows with model layers, KV heads, head dimension, precision, and prompt length. Network topology and locality matter. A deployment that silently falls back from an intended high-speed transport to TCP can have worse TTFT and throughput than an aggregated baseline.

Validate:

• bytes transferred by model and input-length bucket;
• transfer queue and duration;
• transport selected in production;
• compatibility of model revision, tensor parallelism, block size, and KV format;
• cleanup after cancellation and worker failure;
• backpressure when decode capacity is unavailable.

NVIDIA Dynamo documentation explicitly treats fast KV transfer as an early validation item and warns that transfer can dominate performance. This is the central engineering boundary, not a final optimization detail.

Routing and scheduling

The router coordinates two capacity pools and should optimize for SLO goodput, not only shortest queue.

prefill_score =
  estimated_prefill_time(input_tokens, worker_profile)
  + prefill_queue_delay

decode_score =
  active_sequences
  + expected_output_pressure
  + kv_transfer_cost(prefill_worker, decode_worker)

Useful routing inputs include:

• input and expected output length;
• tenant priority and latency SLO;
• prefix or KV locality;
• worker queue, memory, and failure state;
• transfer topology;
• model and adapter compatibility.

Do not trust model-provided length predictions as a hard allocation fact. Use request limits, historical distributions, and runtime budget enforcement.

Aggregated versus disaggregated

Start with an optimized aggregated baseline using continuous batching and prefix caching. Add disaggregation only when phase interference or scaling asymmetry is measured.

Relationship to other inference optimizations

Disaggregation complements but does not replace:

• continuous batching: schedules active sequences efficiently within workers;
• prefix caching: reuses KV state for repeated prompt prefixes;
• KV-aware routing: chooses workers based on cache locality;
• chunked prefill: interleaves prompt chunks with active decode work;
• speculative decoding: verifies draft tokens to reduce decode steps;
• quantization and parallelism: change model compute and memory profiles.

Each mechanism can change whether disaggregation remains beneficial. For example, effective prefix caching reduces prefill work, while speculative decoding changes decode demand. Re-run capacity models after enabling either.

Capacity planning

Measure request distributions before choosing worker ratios:

prefill_load = requests_per_second * average_prefill_seconds
decode_load  = requests_per_second * average_decode_seconds

minimum_prefill_workers ≈ prefill_load / target_utilization
minimum_decode_workers  ≈ decode_load / target_utilization

This is only a starting point. Use input/output length buckets and tail latency because averages hide long-context bursts. Benchmark multiple xP:yD ratios under arrival traces, failures, and SLO constraints.

Track:

• TTFT p50/p95/p99 by input length;
• ITL or time per output token by concurrency;
• request throughput and SLO-qualified goodput;
• prefill/decode queue time and utilization;
• KV transfer duration, bytes, and error rate;
• cancellation cleanup and memory leakage;
• cost per successful request.

Failure handling

Disaggregation adds partial failure states:

Do not blindly redo prefill after every error. Retrying can double GPU work, leak KV allocations, or produce duplicate streamed output. Define request identity and ownership of cleanup.

Benchmark plan

Compare aggregated and disaggregated deployments using the same:

• model revision, quantization, tokenizer, and sampling settings;
• input/output length distributions;
• concurrency and arrival process;
• prefix-cache policy;
• hardware count and cost accounting;
• latency SLO and cancellation behavior.

Run steady state, burst, long-prompt, decode-heavy, worker-loss, and network-degradation scenarios. Report SLO-qualified goodput rather than only maximum tokens per second. A configuration that produces more tokens while violating TTFT or ITL targets is not an improvement for that service.

Production decision checklist

Adopt disaggregation when:

1. phase-level traces show prefill/decode interference;
2. the two phases need materially different scaling or parallelism;
3. fast, observable KV transfer is available;
4. the router can enforce compatibility and backpressure;
5. failure cleanup and cancellation are tested;
6. benchmarked SLO goodput or cost improves over the baseline.

Remain aggregated when the architecture adds complexity without a measured SLO benefit.

The KV-cache inference guide explains the state being transferred. The broader LLM inference guide covers batching, memory, and serving metrics.

FAQ

Can prefill and decode use different GPU types?

Potentially, if the serving stack supports compatible model state and transfer. Hardware heterogeneity adds scheduling, numerical compatibility, capacity, and operational complexity, so benchmark the exact configuration.

Does disaggregation reduce TTFT?

It can reduce interference and queueing, but KV transfer adds latency. TTFT improves only when saved prefill contention exceeds routing and transfer overhead.

Does it require RDMA?

Not conceptually, but high-volume cross-node KV movement often needs RDMA or an equivalent fast fabric to remain competitive. Validate the actual transport and fallback behavior.

How does prefix caching affect the design?

Prefix caching can reduce prefill compute and favor routing to workers with existing KV state. It may reduce the need for remote prefill or make KV-aware placement more valuable.

Is vLLM disaggregated prefilling production-ready for every workload?

No. Support, topology, connector behavior, and operational maturity vary by version. Follow the deployed version's documentation and benchmark it; do not infer universal readiness from feature presence.

Sources

• NVIDIA Dynamo Disaggregated Serving, accessed 2026-07-28.
• NVIDIA Dynamo design documentation, accessed 2026-07-28.
• vLLM Disaggregated Prefilling, accessed 2026-07-28.