Ollama vs vLLM vs llama.cpp: Local LLM Deployment Guide

Direct Answer

Ollama, vLLM, and llama.cpp overlap, but they optimize different deployment jobs. Choose Ollama for a managed local-model workflow, vLLM for scheduler-driven GPU serving and distributed scale, and llama.cpp for GGUF portability and low-level runtime control. None is a universal winner: the correct choice is the smallest operational stack that passes your model-quality, latency, throughput, hardware, security, and API-contract gates.

This comparison owns the workload-selection and evaluation question. For Modelfiles, native APIs, structured output, and model lifecycle details, use the Ollama deployment guide.

The Three Layers You Are Actually Comparing

A local LLM deployment is a chain of contracts, not just an inference binary. Comparing framework names without controlling the chain usually produces a benchmark of different models, templates, or quantizers.

The layers are:

1. Model contract: weight revision, tokenizer, chat template, adapter, quantization, and context policy.
2. Execution runtime: CPU/GPU backends, kernels, memory placement, and device topology.
3. Serving scheduler: queueing, batching, admission, preemption, and KV cache allocation.
4. Protocol surface: native APIs, compatible APIs, streaming, usage accounting, cancellation, and errors.
5. Operational boundary: authentication, TLS, rate limits, tenant isolation, observability, and rollback.
6. Workload contract: prompt/output distributions, arrival pattern, quality target, and service-level objectives.

This decomposition prevents two common mistakes: treating GGUF and a higher-precision Hugging Face checkpoint as the same model, and treating core token generation speed as end-to-end service capacity.

Ollama vs vLLM vs llama.cpp at a Glance

The practical distinction is operational intent, not a hard user-count threshold.

Ollama is not universally sequential, and llama.cpp is not limited to a custom single-user wrapper. Current official documentation describes parallel processing in Ollama and parallel decoding plus continuous batching in llama-server. Conversely, vLLM's richer scheduler does not guarantee better latency or economics for every model and offered load.

Choose by Workload, Not by a Concurrency Slogan

The right runtime follows from the dominant constraint and the operational team that will own it.

Do not convert this table into rules such as “five users means vLLM.” Five long-context requests can require more memory than hundreds of short requests, while an arrival rate below service capacity may never exercise batching. Measure tokens and time, not labels such as “user.”

Concurrency, Queueing, and KV Cache

Concurrency consumes resident token state, so context length and parallel requests must be planned together. The KV cache capacity guide explains why model weights alone cannot predict serving capacity.

Ollama

Ollama's FAQ documents parallel request processing when memory permits. OLLAMA_NUM_PARALLEL limits parallel requests per model, and memory demand scales with parallelism multiplied by context length. OLLAMA_MAX_QUEUE bounds queued work before overload rejection. Ollama can also keep multiple models loaded and may spread a model across available GPUs when it does not fit on one GPU.

Those capabilities do not imply the same scheduler or distributed execution model as vLLM. Observe ollama ps, request timings, rejection behavior, processor placement, and actual resident memory on the target host.

vLLM

vLLM combines a serving scheduler with paged KV-cache allocation. Its parallelism guide distinguishes single-GPU execution, tensor parallelism within a node, and tensor plus pipeline parallelism across nodes. Startup logs report available KV-cache capacity and estimated maximum concurrency, but those estimates are capacity signals rather than an application SLO.

PagedAttention's original paper reports large gains over the systems and workloads evaluated in that paper. It does not establish a timeless multiplier between current vLLM and current Ollama. Treat paging as a mechanism that reduces fragmentation and duplication, then measure its outcome under your sequence distribution.

llama.cpp

llama-server documents parallel decoding, multiple users, slots, continuous batching, cache controls, metrics, API keys, and multi-device split modes. llama-bench separately measures prompt processing and text generation and explicitly excludes tokenization and sampling time. Use it to isolate runtime changes, not to claim HTTP service latency.

Model and Quantization Equivalence Comes First

A fast response from a different model is not a serving optimization. Before comparing runtimes, pin:

• source model and immutable weight revision;
• tokenizer files and revision;
• chat template and special tokens;
• adapter and merged-weight state;
• quantization method, calibration data, and artifact checksum;
• context and rope settings;
• generation defaults, stop rules, seed, and structured-output policy.

Ollama and llama.cpp commonly consume GGUF artifacts, while vLLM commonly serves other repository formats and quantization schemes. Even when artifacts descend from the same base checkpoint, quantization can change instruction following, tool calls, structured output, long-context behavior, and token probabilities. Run a task-level quality suite before comparing speed.

The model quantization guide covers format and precision trade-offs. Do not use a universal “quality retained” percentage: the result depends on the model, quantizer, calibration, task, and metric.

OpenAI Compatibility Is a Testable Contract

“OpenAI-compatible” identifies a protocol family, not complete behavioral equivalence. Ollama says it supports parts of the OpenAI API. vLLM documents supported APIs, unsupported or ignored parameters, chat-template requirements, and engine-specific fields. llama-server also documents compatible routes, but its supported surface and semantics must be checked independently.

Build a migration contract that covers:

Changing only base_url is acceptable after this suite passes. It is not evidence that prompts, outputs, accounting, or errors are equivalent.

A Reproducible Benchmark Contract

A useful local LLM benchmark reproduces the production arrival process and reports both latency and useful completed work. Pin this manifest with every result:

benchmark:
  hardware:
    accelerator: "exact model and count"
    interconnect: "PCIe, NVLink, or none"
    cpu_ram_storage: "record exact configuration"
  software:
    os_driver_runtime: "immutable versions"
    engine_revision: "tag or commit"
  model:
    source_revision: "immutable revision"
    tokenizer_revision: "immutable revision"
    artifact_sha256: "checksum"
    quantization: "method and settings"
    chat_template_sha256: "checksum"
  workload:
    prompt_tokens: "distribution, not only mean"
    output_tokens: "distribution, not only maximum"
    prefix_reuse: "distribution"
    request_rate: "requests per second and arrival model"
    max_concurrency: "client-side cap"
    warmup_requests: "recorded count"
  generation:
    temperature: 0
    seed: 42
    stop_policy: "pinned"
  slos:
    ttft_ms_p95: "application target"
    itl_ms_p95: "application target"
    e2e_ms_p95: "application target"

Measure at least:

• TTFT, inter-token latency or TPOT, and end-to-end latency at p50/p95/p99;
• request and output-token throughput;
• success, timeout, cancellation, and overload-rejection rates;
• SLO-qualified goodput;
• queue time, Prefill, Decode, and preemption where available;
• model, KV-cache, activation, host-memory, and device-memory use;
• task quality and structured-output validity;
• startup, model load, and recovery time as separate cold-path metrics.

vLLM's bench serve command can control request rate, burstiness, maximum concurrency, warmup, prompt/output lengths, percentile metrics, and goodput SLOs. Ollama's native generation response separates model-load, prompt-evaluation, and token-generation durations. llama-bench separates prompt processing from generation but omits tokenization and sampling. These instruments have different boundaries, so a shared external load generator is still required for end-to-end comparison.

Calculate SLO-Qualified Goodput

Goodput counts completed requests that satisfy the application SLO, rather than rewarding a server for accepting work that finishes too late. The following dependency-free script reads normalized JSONL traces.

from __future__ import annotations

import json
import sys
from pathlib import Path


REQUIRED = ("ok", "ttft_ms", "itl_p95_ms", "e2e_ms")


def load_traces(path: Path) -> list[dict]:
    traces = []
    with path.open(encoding="utf-8") as handle:
        for line_number, line in enumerate(handle, start=1):
            if not line.strip():
                continue
            try:
                trace = json.loads(line)
            except json.JSONDecodeError as error:
                raise ValueError(f"line {line_number}: invalid JSON") from error
            missing = [key for key in REQUIRED if key not in trace]
            if missing:
                raise ValueError(f"line {line_number}: missing {missing}")
            traces.append(trace)
    if not traces:
        raise ValueError("trace file is empty")
    return traces


def goodput(traces: list[dict], duration_s: float) -> tuple[int, float]:
    if duration_s <= 0:
        raise ValueError("duration_s must be positive")
    passed = sum(
        bool(trace["ok"])
        and float(trace["ttft_ms"]) <= 800
        and float(trace["itl_p95_ms"]) <= 80
        and float(trace["e2e_ms"]) <= 5000
        for trace in traces
    )
    return passed, passed / duration_s


try:
    source = Path(sys.argv[1])
    duration = float(sys.argv[2])
    count, requests_per_second = goodput(load_traces(source), duration)
    print(f"SLO-passing requests: {count}")
    print(f"Goodput: {requests_per_second:.2f} requests/s")
except (IndexError, OSError, ValueError) as error:
    raise SystemExit(f"usage: python goodput.py TRACE.jsonl DURATION_S\n{error}") from error

Normalize the same event boundaries for every engine. A server-native metric can enrich the trace, but it must not silently redefine when arrival, first token, last token, and completion occur. The LLM inference guide explains these measurement boundaries.

Tune Only After the Baseline Is Valid

Optimization should change one controlled dimension at a time.

Ollama tuning order

1. Pin the model and inspect actual processor placement with ollama ps.
2. Set only the context needed by the workload.
3. Increase parallelism while tracking memory, queueing, and rejection.
4. Test Flash Attention and KV-cache type on supported hardware.
5. Set model residency based on measured reuse and load latency.

vLLM tuning order

1. Confirm model fit and startup-reported KV-cache capacity.
2. Establish a request-rate curve before changing scheduler limits.
3. Evaluate chunked prefill when long prompts interfere with decode.
4. Evaluate prefix caching only with a representative prefix-reuse distribution.
5. Choose tensor/pipeline parallelism from model fit and topology, then measure communication overhead.
6. Track queue, KV usage, TTFT, ITL, E2E, preemption, and request success.

llama.cpp tuning order

1. Pin build commit, backend, GGUF checksum, threads, and device placement.
2. Separate prompt processing from token generation with llama-bench.
3. Tune batch, micro-batch, cache type, context, and offload independently.
4. Configure server slots and continuous batching, then run an HTTP workload test.
5. Recheck power, thermals, and sustained performance on edge devices.

Speculative decoding, cache quantization, and aggressive batching are experiments, not guaranteed multipliers. Keep them only when they improve the target goodput without violating quality or tail-latency gates.

Production Security and Operations

Running a model locally changes the data path; it does not automatically satisfy privacy, security, or compliance requirements. A production boundary still needs:

• loopback-only binding for developer machines, or a controlled private network for services;
• gateway authentication, authorization, TLS, request-size limits, and rate limits;
• tenant-aware queues, cache isolation, and log redaction;
• allowlisted model artifacts, checksums, licenses, and provenance;
• health/readiness probes, load shedding, timeouts, cancellation, and graceful draining;
• audit events that do not persist sensitive prompts by default;
• pinned images or commits, staged rollout, and a tested rollback;
• private multi-node traffic, because distributed engine traffic may not be safe on an untrusted network.

Ollama binds locally by default, while changing its bind address can expose an unauthenticated service unless an external boundary is added. llama-server supports API keys, but key validation alone does not provide tenant isolation or transport security. vLLM metrics are valuable, but metric names follow an engine lifecycle and should be pinned to the deployed revision.

Migration and Rollback Gates

A migration is complete only when model behavior, protocol behavior, and operations all pass.

1. Freeze identity: record source revision, tokenizer, template, quantizer, checksum, and generation defaults.
2. Run quality parity: compare task scores, structured output, tool calls, refusals, and long-context behavior.
3. Run API parity: replay successful, malformed, streaming, cancelled, and overload requests.
4. Run load tests: sweep offered load and context/output distributions; locate the SLO saturation point.
5. Validate operations: authentication, metrics, traces, redaction, readiness, draining, restart, and model load.
6. Canary: route a controlled cohort with backend-tagged metrics and no silent fallback.
7. Rollback: preserve the old artifact, configuration, routing rule, and state needed to revert.

The hybrid pattern can be useful, but “Ollama in development and vLLM in production” creates drift unless both environments pin a compatible model contract. A fast local prototype is not a production rehearsal when the template, quantization, or defaults change.

Common Failure Modes

The most expensive deployment failures come from an invalid comparison or an uncontrolled boundary.

Official Sources

• Ollama FAQ — parallel requests, queues, model residency, GPU placement, networking, Flash Attention, and KV-cache configuration.
• Ollama OpenAI compatibility — documented compatible API surface.
• Ollama generate API — native response timing and token-count fields.
• vLLM parallelism and scaling — single-GPU, tensor-parallel, pipeline-parallel, and multi-node boundaries.
• vLLM OpenAI-compatible server — endpoint, parameter, template, and generation-config behavior.
• vLLM production metrics — queue, cache, prefill, decode, TTFT, ITL, E2E, and request signals.
• vLLM bench serve — offered-load, concurrency, percentile, trace, and goodput controls.
• llama.cpp server — parallel decoding, continuous batching, routes, metrics, keys, slots, and device controls.
• llama-bench — core prompt-processing and generation benchmark boundaries.
• PagedAttention paper — paged KV-cache mechanism and original experimental scope.

FAQ

Is vLLM always faster than Ollama?

No. vLLM targets high-throughput accelerator serving, but observed performance depends on the exact model, quantization, hardware, request lengths, offered load, and SLO. Ollama may have lower operational cost for a local workflow. Compare quality-qualified goodput and tail latency, not an isolated tokens-per-second number.

Can Ollama process requests in parallel?

Yes. Ollama documents per-model parallel request processing through OLLAMA_NUM_PARALLEL when memory permits, plus a bounded request queue. Parallelism multiplies context-related memory, so test the exact model, context, and host instead of assuming a fixed concurrency ceiling.

Can llama.cpp serve multiple users?

Yes. The official llama-server supports parallel slots and continuous batching. Use llama-bench to isolate core runtime performance and a separate streaming HTTP load test to measure queueing, TTFT, cancellation, errors, and end-to-end multi-user behavior.

Can an OpenAI client move between Ollama and vLLM unchanged?

Only after a conformance suite passes. Compatible endpoints do not guarantee identical parameter support, chat templates, model names, defaults, stream events, usage accounting, or errors. Pin those behaviors and treat a base-URL switch as the final routing step, not the migration plan.

Which metrics decide a local LLM serving winner?

Use task quality as a hard gate, then compare TTFT, ITL or TPOT, E2E latency, output-token throughput, errors, queueing, memory, and SLO-qualified goodput across realistic offered loads. Include cold-start and recovery separately. The winner is the system that meets the workload SLO with acceptable cost and operator burden.

Summary

Ollama, vLLM, and llama.cpp are three different operational answers to local LLM inference. Start with workload, model identity, hardware, API, and security contracts; then measure realistic arrival rates and SLO-qualified goodput. Keep Ollama when its integrated workflow and measured capacity satisfy the service, choose vLLM when scheduling and distributed GPU serving provide verified value, and choose llama.cpp when portability and runtime control dominate. Re-evaluate with pinned artifacts and traces whenever the model, context, quantization, engine, or hardware changes.