Ollama: Local Models, Cloud APIs, and Production Boundaries

Cloud APIs remove inference infrastructure work; local runtimes give a team direct control over selected model artifacts and where inference runs. Ollama now supports both paths, so the product name alone does not identify the execution location. Local disks, logs, model downloads, operators, tools, backups, cloud models, and remote fallbacks all belong in the data-flow and threat model.

Direct answer: Ollama is a developer-oriented model runtime and distribution layer for supported large language models (LLMs). It can execute downloaded models through a local server, route signed-in clients to cloud models, or serve direct authenticated requests at ollama.com. Its CLI, Modelfiles, native /api, and compatibility endpoints simplify integration, but they do not remove the need to decide execution location, identity, authorization, data handling, capacity, and rollback.

Separate Local and Cloud Execution

Ollama exposes similar request shapes across different trust boundaries. Record the selected path explicitly:

The local daemon does not validate API keys. A placeholder key required by an SDK is not an access control. Direct cloud APIs require an Ollama API key, which must stay out of browser code, source control, and logs. Compatibility endpoints implement documented subsets, so test the exact fields, streaming events, tool behavior, and errors your client uses. Teams with a strict local-only requirement should disable cloud features and verify egress, rather than assuming a local base URL makes every selected model local.

How the Local Runtime Works

For a local model, Ollama resolves an artifact, loads it through a supported backend, applies its prompt template and runtime options, then streams or returns the generated response through an API. A cloud model follows a remote data path instead. In both cases, the surrounding application remains responsible for identity, policy, validation, persistence, and any business action.

A request passes through five operational stages:

1. Artifact resolution: Ollama locates a pulled model tag or a supported local model or adapter declared by FROM and ADAPTER.
2. Model loading: The runtime allocates model weights and context on CPU, GPU, or a split supported by the current platform.
3. Prompt construction: The model template combines the system message, conversation, and runtime parameters.
4. Inference: /api/generate handles a prompt-oriented request; /api/chat handles message history and can include tools or a structured-output schema.
5. Result handling: The final response includes counters and durations that the application can record. Generated text and tool arguments remain untrusted input.

The native API is the clearest contract when you control the integration:

Build Reproducible Models with a Modelfile

A Modelfile records the base artifact, minimum runtime, prompt behavior, adapters, and inference parameters that must travel with an experiment. It improves reproducibility, but it does not enforce authorization or guarantee that generated data is valid.

This code-review example deliberately separates prompt guidance from output enforcement:

# Replace these values with versions and artifacts your team has tested.
REQUIRES 0.14.0
FROM llama3.2:3b

PARAMETER temperature 0
PARAMETER num_ctx 4096
PARAMETER num_predict 512

SYSTEM """
Review the supplied code for bugs, security defects, and performance risks.
Return concise findings. The calling application defines and validates the
required JSON Schema; do not trigger tools or perform external actions.
"""

Save it as CodeReviewer.modelfile, create the local model, and inspect the resolved configuration:

ollama create code-reviewer -f ./CodeReviewer.modelfile
ollama show --modelfile code-reviewer
ollama run code-reviewer

For a controlled environment, record at least the Ollama version, model tag, resolved Modelfile, artifact provenance and license, host and driver details, and test-set revision. A moving model tag or runtime upgrade is a deployment change even if application code is unchanged.

Importing GGUF Models and Adapters

Ollama can serve supported GGUF files and adapters; it is not a general training framework. Train or fine-tune with an appropriate framework, export a supported artifact, and then verify conversion, tokenizer, prompt template, license, and quality before importing it.

FROM ./my-reviewed-model.gguf
# The adapter must match the base model used during training.
ADAPTER ./my-reviewed-adapter.gguf

An adapter built for a different base model can behave incorrectly. Keep the base artifact, adapter, conversion command, and evaluation result together in the model release record.

Use Structured Outputs Without Trusting the Model

Ollama structured outputs can constrain a native chat or generate response to JSON or a supplied JSON Schema. Schema-conforming output is easier to parse, but it can still contain false, unsafe, or unauthorized values; validate both shape and business meaning.

The native endpoint accepts a JSON Schema in format:

curl -s http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "code-reviewer",
    "messages": [{
      "role": "user",
      "content": "Review: function add(a, b) { return a - b; }"
    }],
    "stream": false,
    "format": {
      "type": "object",
      "properties": {
        "issues": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "type": {"enum": ["bug", "security", "performance"]},
              "description": {"type": "string"}
            },
            "required": ["type", "description"]
          }
        }
      },
      "required": ["issues"]
    },
    "options": {"temperature": 0}
  }'

Use one Go contract for the request schema and response validation. This standard-library example reads OLLAMA_API_KEY only when the selected endpoint needs a Bearer token:

package main

import (
	"bytes"
	"context"
	"encoding/json"
	"flag"
	"fmt"
	"io"
	"net/http"
	"os"
	"strings"
	"time"
)

const reviewSchema = `{
  "type": "object",
  "properties": {
    "issues": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "type": {"enum": ["bug", "security", "performance"]},
          "description": {"type": "string"}
        },
        "required": ["type", "description"]
      }
    }
  },
  "required": ["issues"]
}`

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

type ChatRequest struct {
	Model    string          `json:"model"`
	Messages []Message       `json:"messages"`
	Stream   bool            `json:"stream"`
	Format   json.RawMessage `json:"format"`
	Options  map[string]any  `json:"options"`
}

type ChatResponse struct {
	Message Message `json:"message"`
}

type Issue struct {
	Type        string `json:"type"`
	Description string `json:"description"`
}

type Review struct {
	Issues []Issue `json:"issues"`
}

func requestReview(
	ctx context.Context,
	client *http.Client,
	endpoint string,
	model string,
) (ChatResponse, error) {
	payload := ChatRequest{
		Model: model,
		Messages: []Message{{
			Role:    "user",
			Content: "Review: function add(a, b) { return a - b; }",
		}},
		Stream:  false,
		Format:  json.RawMessage(reviewSchema),
		Options: map[string]any{"temperature": 0},
	}
	body, err := json.Marshal(payload)
	if err != nil {
		return ChatResponse{}, err
	}
	request, err := http.NewRequestWithContext(
		ctx, http.MethodPost, endpoint, bytes.NewReader(body),
	)
	if err != nil {
		return ChatResponse{}, err
	}
	request.Header.Set("Content-Type", "application/json")
	if key := strings.TrimSpace(os.Getenv("OLLAMA_API_KEY")); key != "" {
		request.Header.Set("Authorization", "Bearer "+key)
	}

	response, err := client.Do(request)
	if err != nil {
		return ChatResponse{}, err
	}
	defer response.Body.Close()
	if response.StatusCode < 200 || response.StatusCode >= 300 {
		message, _ := io.ReadAll(io.LimitReader(response.Body, 4096))
		return ChatResponse{}, fmt.Errorf(
			"ollama returned %s: %s",
			response.Status,
			strings.TrimSpace(string(message)),
		)
	}
	var result ChatResponse
	if err := json.NewDecoder(response.Body).Decode(&result); err != nil {
		return ChatResponse{}, err
	}
	return result, nil
}

func validateReview(content string) (Review, error) {
	var review Review
	if err := json.Unmarshal([]byte(content), &review); err != nil {
		return Review{}, fmt.Errorf("invalid review JSON: %w", err)
	}
	if len(review.Issues) > 20 {
		return Review{}, fmt.Errorf("too many issues: %d", len(review.Issues))
	}
	allowed := map[string]bool{
		"bug": true, "security": true, "performance": true,
	}
	for index, issue := range review.Issues {
		if !allowed[issue.Type] || strings.TrimSpace(issue.Description) == "" {
			return Review{}, fmt.Errorf("invalid issue at index %d", index)
		}
	}
	return review, nil
}

func main() {
	endpoint := flag.String("endpoint", "", "Ollama chat endpoint")
	model := flag.String("model", "code-reviewer", "model name")
	timeout := flag.Duration("timeout", 30*time.Second, "request timeout")
	flag.Parse()

	response := ChatResponse{
		Message: Message{
			Content: `{"issues":[{"type":"bug","description":"add subtracts b"}]}`,
		},
	}
	if *endpoint != "" {
		ctx, cancel := context.WithTimeout(context.Background(), *timeout)
		defer cancel()
		var err error
		response, err = requestReview(
			ctx, &http.Client{Timeout: *timeout}, *endpoint, *model,
		)
		if err != nil {
			fmt.Fprintln(os.Stderr, err)
			os.Exit(1)
		}
	}

	review, err := validateReview(response.Message.Content)
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	output, err := json.MarshalIndent(review, "", "  ")
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	fmt.Println(string(output))
}

Run it without flags to exercise the offline validation fixture, or pass -endpoint http://localhost:11434/api/chat to call a local server. For direct cloud access, pass -endpoint https://ollama.com/api/chat, select a hosted model with -model, and inject OLLAMA_API_KEY from a secret manager. Then apply explicit rules such as allowed line ranges, authorization checks, and human approval. OWASP classifies unsanitized model output passed to shells, browsers, SQL, file paths, or privileged functions as improper output handling.

Choose the Native API or OpenAI Compatibility Deliberately

Ollama's /v1 surface supports documented parts of the OpenAI API; it is an interoperability adapter rather than a universal drop-in replacement. Prefer /api for Ollama-specific lifecycle controls and use /v1 when the exact client operation has been tested.

Do not make a production switch by changing only baseURL. Build contract tests for the operations you use: message roles, streaming chunks, structured output, tools, usage fields, cancellation, timeout, retry, and error mapping.

Measure Context, Model Residency, and Request Cost

Ollama capacity depends on model weights, quantization, context length, concurrency, backend, and device placement. A larger context consumes more memory, and an apparently available model may be partly offloaded to CPU, so measure the deployed combination instead of relying on a generic hardware table.

Use ollama ps to inspect active placement and context:

ollama ps

For API requests, record the final non-streaming response or aggregate the final streaming event. Native responses expose fields including total_duration, load_duration, prompt_eval_count, prompt_eval_duration, eval_count, and eval_duration. Durations are reported in nanoseconds.

curl -s http://localhost:11434/api/generate \
  -d '{
    "model": "code-reviewer",
    "prompt": "Explain one risk of executing model-generated shell commands.",
    "stream": false,
    "keep_alive": "5m"
  }' |
jq '{
  total_ms: (.total_duration / 1000000),
  load_ms: (.load_duration / 1000000),
  prompt_tokens: .prompt_eval_count,
  output_tokens: .eval_count,
  output_tokens_per_second:
    (if .eval_duration > 0
     then (.eval_count / (.eval_duration / 1000000000))
     else null end)
}'

Benchmark complete tasks rather than a single warm request. Separate cold model load from prompt evaluation and token generation, and report P50/P95 latency, failure and timeout rates, memory pressure, processor placement, quality, and concurrent-user behavior. Use keep_alive: 0 when a test must unload the model immediately; use a bounded duration only after measuring the memory trade-off.

Secure an Ollama Service Before Network Exposure

The safest default is loopback access behind an application boundary. Binding Ollama beyond localhost turns a developer runtime into a reachable model service; add controls before changing the bind address.

Do not set OLLAMA_ORIGINS="*" as a general production fix. Browser origin policy is not authentication, and a private LAN is not an authorization system.

Ollama vs LM Studio vs vLLM

The right runtime follows the operating model, not a universal speed ranking. Benchmark the same artifact and workload when two runtimes support it.

Choose Ollama when local workflow simplicity and artifact control matter more than maximizing shared-serving throughput. Evaluate vLLM or another serving platform when batching, accelerator utilization, large concurrency, autoscaling, or mature multi-tenant controls are primary requirements.

Production Readiness and Rollback Checklist

A successful local demo proves only that one request ran in one environment. Promotion to a shared service requires reproducible artifacts, workload evidence, security controls, and a rollback path.

1. Pin and record the Ollama release, model tag, resolved Modelfile, artifact source, and license.
2. Evaluate representative prompts, adversarial inputs, malformed output, refusals, and domain-specific quality.
3. Test cold and warm latency, context sizes, concurrency, memory pressure, timeouts, cancellation, and recovery.
4. Keep Ollama private; enforce identity, tenant authorization, quotas, body limits, and audit in the surrounding service.
5. Validate model output and tool arguments before rendering, storing, executing, or forwarding them.
6. Define data retention, log redaction, egress, backup, incident response, and model-removal procedures.
7. Keep the previous runtime, model artifact, configuration, and evaluation result available for rollback.

Frequently Asked Questions

Does Ollama work without a GPU?

Some model and platform combinations can run on CPU. Whether the result is useful depends on model size and quantization, context, concurrency, backend, target latency, and quality. Test the exact workload and inspect processor placement with ollama ps.

Is Ollama private by default?

Requests to a loopback endpoint can be processed locally, but that does not prove end-to-end privacy. Model downloads, cloud models, application telemetry, tools, plugins, logs, backups, and remote fallbacks can create other data paths. Document and test the complete flow.

Can Ollama replace vLLM?

They overlap but optimize for different operating models. Ollama emphasizes local developer experience and model management; vLLM emphasizes throughput-oriented serving. Compare them with the same model, context, concurrency, hardware, and quality criteria.

Does JSON Schema make model output safe?

No. It constrains response shape and improves parsing. It does not prove factual accuracy, authorization, safe HTML or SQL, allowed file paths, or permission to invoke a tool. Apply normal validation and security controls after parsing.

Conclusion

Ollama makes local model evaluation and cloud model access approachable, but neither path is a complete production architecture. Record where inference runs, keep local daemons private, protect direct cloud keys, use Modelfiles for reproducibility, validate structured output, measure the real workload, and retain a tested rollback path.

Official Sources

• Ollama API introduction — native API base URL and request examples
• Ollama API authentication — local no-auth boundary and direct cloud Bearer authentication
• Ollama Cloud — direct cloud access, model naming, data handling, and local-only controls
• Ollama Modelfile reference — supported instructions and parameters
• Ollama structured outputs — JSON mode, JSON Schema, Pydantic, and Zod validation examples
• Ollama context length — context configuration, memory trade-offs, and ollama ps
• Ollama OpenAI compatibility — documented compatibility surface and examples
• Ollama Anthropic compatibility — local and hosted /v1/messages behavior
• Ollama FAQ — local access, networking, cloud models, and operational configuration
• OWASP LLM05: Improper Output Handling — validation and contextual handling of model-generated output

Further Reading

• Ollama — Ollama glossary entry and core concepts
• Local LLM Deployment: Ollama vs vLLM Performance Optimization — Production benchmarks comparing Ollama and vLLM
• KV Cache for LLM Inference — Why context length, concurrency, and cache precision change inference memory
• Small Language Models Edge Deployment — Deploying lightweight models with Ollama on edge devices
• Model Quantization Complete Guide — Understanding GGUF, GPTQ quantization formats used with Ollama