What is Cold Start?
Cold Start is the startup interval triggered when ready model-serving capacity is unavailable, measured from a declared demand event until a new or restored replica passes readiness or returns its first valid inference response.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Lifecycle and Readiness Boundary
Record monotonic milestones for demand observed, capacity assigned, image available, weights verified and resident, distributed workers joined, runtime initialized, representative warmup passed, endpoint registered and first valid response. Readiness should exercise the model path needed by production rather than only checking a TCP socket. For sharded models, no worker is ready until the required rank set and communication topology are healthy. Failed attempts and retries remain part of the cold-path outcome.
Mitigation and Tradeoffs
Minimum replicas and warm pools avoid the cold path at an idle-capacity cost. Smaller or quantized artifacts reduce transferred bytes only if they satisfy the same quality and compatibility contract. Node-local weight caches, parallel verified downloads, prebuilt kernels, staged initialization and predictive scale-out target different phases. Checkpoint and restore can move work off the request path, but artifacts bind runtime, driver, hardware and security state; [NVIDIA Dynamo Snapshot](https://developer.nvidia.com/blog/nvidia-dynamo-snapshot-fast-startup-for-inference-workloads-on-kubernetes) describes one implementation with explicit limitations rather than a universal shortcut.
Measurement and Capacity Planning
Measure cold and warm populations separately and report p50, p95, p99, failure rate and phase durations by model artifact, region, node type, storage path, runtime revision and trigger. Include queue growth while capacity initializes, because a faster boot that admits traffic too early can worsen user-visible tail latency. Replay realistic idle gaps and bursts, then choose a warm-capacity floor and scale lead time from the product SLO and total cost. The [Google Cloud GPU guidance](https://docs.cloud.google.com/run/docs/configuring/services/gpu-best-practices) is useful for its platform but its exact controls and timings do not transfer automatically.
Key Characteristics
- Begins with demand for unavailable ready capacity and ends at an explicitly defined readiness event
- Can include placement, image, weight, accelerator, runtime, warmup, health, and registration phases
- Occurs during scale-from-zero, scale-out, rollout, failover, or worker replacement
- Differs from ordinary TTFT, cold prefix cache, and steady-state inference latency
- Trades idle warm-capacity cost against tail latency and burst resilience
- Must be analyzed as a distribution with failed starts and queued requests
Common Use Cases
- Setting readiness probes that wait for a representative inference warmup
- Choosing minimum replicas or warm-pool capacity for an interactive model
- Comparing remote, node-local, shared-memory, or snapshot-based weight loading
- Measuring rollout and failover capacity recovery without routing traffic early
- Separating startup delay from steady-state TTFT and decode performance
Example
Loading code...Frequently Asked Questions
Is a Cold Start the same as TTFT?
No. Cold Start describes capacity moving from unavailable to ready. TTFT measures one request from its defined start until the first output token. A request routed to a cold replica may include startup inside TTFT, but warm requests have TTFT without a Cold Start. Report both boundaries instead of treating them as synonyms.
When is an AI model replica truly ready?
It is ready only after required weights are verified and resident, workers and collectives are healthy, runtime initialization and representative warmup have passed, and routing registration is complete. A running container or open port can precede model readiness and should not receive normal traffic.
How can LLM Cold Starts be reduced?
Measure the dominant phase first. Use warm capacity for provisioning delay, smaller verified images for pulls, local or parallel storage for weights, compatible precompiled artifacts for runtime setup, and checkpoint or snapshot restore where supported. Each optimization needs a cold-path benchmark and rollback plan.
Should an inference service scale to zero?
Scale-to-zero is appropriate when idle cost matters more than startup delay, such as asynchronous or infrequent jobs. Interactive services need a warm floor when the cold-start distribution cannot meet their SLO. Make the decision from arrival patterns, queue tolerance, recovery needs and total cost rather than a universal rule.
How should Cold Start performance be benchmarked?
Trigger genuine cold states repeatedly and record every milestone, retries and failures. Report percentile distributions by artifact, region, hardware, storage and trigger, plus queue growth and first-valid-response latency. Do not mix warm requests into the same average or compare systems with different readiness definitions.