What is Disaggregated Serving?

Disaggregated Serving is an LLM inference architecture that runs prompt prefill and token decoding in separate worker pools connected by a key-value cache transfer path.

Quick Facts

CreatedDeveloped in modern LLM serving systems in the 2020s
SpecificationOfficial Specification

How It Works

Prefill processes input tokens and creates the initial KV cache, while decode repeatedly reads that state to generate output tokens. Their compute, memory, batching, and latency profiles differ, so disaggregated serving lets operators size and scale each phase independently and reduce interference between long prompts and active generations. Every request must move or expose its KV state from a prefill worker to a decode worker, making transfer bandwidth, locality, routing, compatibility, and failure recovery part of the critical path. The architecture can improve service-level objective isolation, but it is not an automatic throughput optimization and should be benchmarked against an aggregated baseline. See the <a href="https://qubittool.com/blog/disaggregated-llm-serving-prefill-decode">disaggregated LLM serving guide</a> for routing and capacity planning.

Key Characteristics

  • Separates prefill and decode into independently scalable worker pools
  • Transfers or remotely exposes KV-cache state between phases
  • Can isolate time to first token from inter-token latency pressure
  • Supports phase-specific parallelism, batching, and hardware choices
  • Adds routing, network, compatibility, observability, and recovery complexity

Common Use Cases

  1. Serving long-prompt and long-generation workloads with different bottlenecks
  2. Scaling prefill and decode capacity independently
  3. Protecting active decode streams from bursty prompt processing
  4. Applying KV-aware routing across specialized worker pools
  5. Benchmarking phase-specific hardware and parallelism under explicit SLOs

Example

loading...
Loading code...

Frequently Asked Questions

Why separate prefill and decode?

They have different compute, memory, batching, and latency behavior. Separate pools can scale each bottleneck independently and reduce interference.

Does disaggregated serving always improve throughput?

No. KV transfer, routing, synchronization, and underused workers can outweigh the benefits. Benchmark the real workload against an aggregated deployment.

Why is KV-cache transfer important?

Decode cannot continue without the state produced during prefill. Slow transfer can dominate time to first token and erase the expected isolation benefit.

Is disaggregated serving the same as speculative decoding?

No. Disaggregation places prefill and decode on different workers. Speculative decoding accelerates token generation by verifying draft tokens.

When should aggregated serving remain the default?

It is often simpler for short prompts, small models, low concurrency, limited networking, or workloads without clearly different phase bottlenecks.

Related Tools

AI Websites Directory

An authoritative, comprehensive, and continuously updated AI resources directory. It covers global and domestic model providers, open-source ecosystems, research indexes and leaderboards, developer platforms, and curated tool catalogs—helping you quickly discover, compare, and choose the right AI products and references. Supports keyword search and favorites, with clear category sections and an expanding dataset for better experience.

JSON Formatter

Format, beautify, validate and minify JSON online for free. Features syntax highlighting, tree view, history tracking, and one-click copy. No signup required. 100% client-side processing for privacy.

Related Terms

Prefill

Prefill is the LLM inference phase that processes the full input prompt in parallel and produces the initial key-value cache before token-by-token decoding begins.

Decode Phase

Decode Phase is the LLM inference stage that generates output one token at a time using the KV cache created during prefill.

KV Cache

KV Cache (Key-Value Cache) is inference state that stores the attention Key and Value tensors already computed for prior tokens, so autoregressive decoding can reuse them instead of projecting the full prefix again.

Continuous Batching

Continuous Batching is an LLM serving technique that dynamically groups active requests during inference, adding new requests and removing completed ones without waiting for a fixed batch to finish.

Related Articles

Disaggregated LLM Serving: Prefill and Decode

Learn when to separate LLM prefill and decode into independent worker pools. Understand KV-cache transfer, TTFT and ITL isolation, routing, tuning, and failure modes.

2026-07-28

LLM Inference Complete Guide [2026]: From Tokenization and KV Cache to Text Generation

Learn how Large Language Models generate text. A deep dive into the LLM inference process, covering tokenization, Prefill vs. Decode phases, KV Cache optimization, and latency metrics.

2026-04-07

Local LLM Deployment 2026: Ollama vs vLLM Tuning

2026 benchmarks show vLLM delivers 16x throughput over Ollama at scale. Compare both with tuning strategies for PagedAttention, quantization, and multi-GPU.

2026-05-16