What is Model Serving?
Model Serving is the production system and operational practice that loads a versioned model artifact, executes inference behind a defined interface, and keeps its behavior available, observable and controllable under real traffic.
How It Works
Promote an Immutable Artifact Contract
Identify every release with immutable digests or revisions for weights, Tokenizer or feature pipeline, Template, Adapter, pre/post-processing code, Runtime image and configuration. Version the request schema, response schema, error taxonomy, streaming events, Finish Reason, Usage accounting and timeout behavior. Validate licenses, provenance, access policy and secret boundaries before load. A model alias is a routing name, not sufficient evidence that two Replicas execute the same artifact.
Separate Liveness, Readiness and Traffic
A live process may still be unable to serve because weights are loading, warmup is incomplete, collective peers are missing or capacity is exhausted. Readiness should prove the intended artifact and dependencies are usable before the endpoint enters routing. [KServe's control plane](https://kserve.github.io/website/docs/concepts/architecture/control-plane) separates lifecycle management from inference execution and supports autoscaling and traffic control. Use Shadow, Canary or blue-green routing, preserve sticky state when required, drain in-flight streams, and keep a tested rollback target.
Scale and Observe Accepted Work
CPU or GPU utilization alone is a lagging and incomplete scaling signal. Observe arrival rate, Queue depth and age, admission rejections, Batch occupancy, Cache pressure, TTFT, TPOT, end-to-end latency, errors, cancellations, model quality, safety and cost by artifact revision. Scale early enough to cover model-load delay and topology constraints, but cap admission so queues cannot grow without bound. [Triton model management](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_management.html) shows why loading state, model readiness and safe repository updates are separate concerns.
Key Characteristics
- Binds model and preprocessing artifacts to a versioned execution and API contract
- Separates control-plane lifecycle, routing and scaling from data-plane inference
- Supports online, streaming, asynchronous or batch request semantics
- Uses readiness, admission, draining and rollback controls beyond process liveness
- Measures latency, Throughput, errors, quality, safety and cost per artifact revision
- Protects model access, tenant data, credentials, logs and intermediate state
Common Use Cases
- Exposing classifiers, Embedding models, rerankers or LLMs through stable APIs
- Promoting a new model and Runtime pair through Shadow and Canary traffic
- Autoscaling expensive accelerators while bounding Queue time and Cold Start
- Routing tenants or tasks to compatible model revisions and capacity pools
- Rolling back weights, Tokenizer, Template and Runtime as one tested contract
Example
Loading code...Frequently Asked Questions
How is Model Serving different from model inference?
Inference is the execution that maps an input to a model output. Model Serving wraps that execution in versioned interfaces, lifecycle management, routing, capacity, observability, security and failure handling so applications can depend on it.
What must be versioned in a serving release?
Version weights, Tokenizer or feature logic, Template, Adapter, pre/post-processing, Runtime image, precision, configuration and API semantics. Rollback is unsafe when it restores only the weights but leaves an incompatible surrounding contract.
What is the difference between liveness and readiness?
Liveness asks whether the process should be restarted. Readiness asks whether this Replica should receive traffic. A process can be live while weights load, warmup runs, peers are missing or the model is unavailable.
Which metrics should a model service monitor?
Monitor demand, Queue depth and age, admission outcomes, latency percentiles, Throughput, errors, cancellations, resource and Cache pressure, cost, output-schema validity, task quality and safety, all segmented by artifact revision.
How should a new model version be rolled out?
Validate it offline, load and warm it without traffic, run Shadow checks, then increase Canary traffic behind automatic quality and SLO gates. Keep the previous complete artifact contract ready, drain stateful requests safely, and test rollback before promotion.