LLM Gateway Architecture: Control Plane and Data Plane

Direct Answer

An LLM Gateway is a policy-enforcing network intermediary between AI applications and model backends. A production design separates a low-latency data plane, which handles identity, protocol translation, quotas, streaming, failure control, and usage events, from a versioned control plane, which publishes routes, backend capabilities, secrets, budgets, and rollout policy. The Gateway reduces duplicated integration work, but it does not make providers equivalent, guarantee model quality, or remove vendor coupling.

This is article #18 in the AI Architect Course. Step-level model selection and output acceptance belong in AI Agent Model Routing; this guide stays at the shared network and policy layer.

Table of Contents

1. When an LLM Gateway earns its place
2. Control plane and data plane
3. Make protocol compatibility explicit
4. Identity, quotas, and admission control
5. Streaming, retries, and circuit breakers
6. Metering, reservation, and reconciliation
7. Tenant isolation, cache safety, and secrets
8. Telemetry without prompt leakage
9. Version, canary, and roll back configuration
10. Validate a Gateway policy before rollout
11. Production evaluation checklist
12. FAQ

When an LLM Gateway earns its place

An LLM Gateway is justified when several applications need the same enforceable boundary, not merely because an application calls an LLM. It centralizes cross-cutting policy that would otherwise drift across SDK wrappers: workload identity, backend credentials, model aliases, capability checks, quotas, egress controls, usage records, and trace propagation.

The decision is operational, not fashionable:

A Gateway changes where coupling lives. Application code can depend on stable aliases such as support-chat, but the Gateway still depends on provider protocols, model behavior, and billing exports. It also becomes a potential outage domain. High availability therefore requires redundant Gateway instances, independent configuration recovery, protected secret access, and a bypass or fail-closed decision for each workload.

Control plane and data plane

The architectural core is a separation between configuration decisions and request execution. The control plane validates and publishes immutable configuration snapshots; the data plane consumes a known-good snapshot without querying a mutable administration database on every request.

Control-plane responsibilities

The control plane owns slow-changing, reviewable state:

• workload identities, tenant boundaries, and authorization rules;
• virtual model aliases and backend candidate sets;
• an endpoint-level capability matrix;
• credential references, never plaintext secrets in route files;
• request, token, concurrency, and budget policies;
• timeout, retry, circuit-breaker, and queue limits;
• cache eligibility, data residency, and telemetry redaction;
• configuration schema, version, signature, activation, and rollback.

Publishing should be transactional. If one route references an absent backend or unsupported capability, reject the whole snapshot instead of letting data-plane replicas observe different partial states.

Data-plane request lifecycle

The data plane performs a deterministic sequence and records the decision:

Authentication answers who is calling. Authorization answers which aliases, tools, data regions, and budget scopes that identity may use. Keep both independent from provider API keys so a compromised application credential cannot directly call every backend.

Make protocol compatibility explicit

A unified endpoint is a contract translator, not proof of semantic equivalence. “OpenAI-compatible” may describe a URL and a subset of JSON fields while leaving tool-call deltas, structured output, reasoning controls, images, audio, token accounting, finish reasons, and error bodies different.

Define capabilities by endpoint and behavior, then test them:

Resolve the virtual alias only after validating required capabilities. If a request requires JSON Schema output, for example, candidates that merely accept an ignored response_format field are not compatible. Rejecting an unsupported contract is safer than silently degrading it.

Provider translation remains versioned. Envoy AI Gateway's release documentation, for example, lists endpoint- and provider-specific translation coverage rather than claiming universal parity. That is the right mental model for any implementation: a tested matrix with known losses.

Identity, quotas, and admission control

Production admission control is multidimensional because request count alone does not bound resource use. A tenant can stay below requests per minute while sending long contexts, opening many streams, or exhausting a shared budget.

Enforce separate rate limits and resource limits at appropriate scopes:

• request rate limits bursts and accidental loops;
• estimated input and maximum output units reserve model capacity before dispatch;
• active concurrency bounds open streams and upstream connections;
• queue depth and queue time preserve latency under saturation;
• budget controls monetary exposure but never substitutes for quality policy;
• provider allocation prevents one tenant from consuming a shared upstream quota.

HTTP 429 Too Many Requests can carry Retry-After, but RFC 6585 deliberately does not define how a server identifies a caller or counts requests. The Gateway must define those semantics. Return a stable machine-readable reason such as tenant_token_reservation_exhausted, the affected scope, and whether retrying later can help.

Reserve before dispatch, settle after completion

For variable-length generation, first reserve against a conservative upper bound derived from the validated request and tenant policy. On completion, settle against actual authoritative usage and release the difference. On cancellation or ambiguous provider failure, move the reservation to a pending state until usage is known or an explicit expiry policy applies.

Do not route to a cheaper or weaker model merely because a budget is exhausted. A budget rejection is honest; a silent model substitution can violate a quality, safety, region, or tool-use contract. Cost-aware model choice belongs inside an approved candidate set whose outputs meet the workload's acceptance gate.

Streaming, retries, and circuit breakers

Streaming changes the failure boundary: once the Gateway has sent response bytes to the client, replaying the request against another backend can duplicate text, tool calls, or side effects. The safest general rule is retry before downstream commitment, never after, unless the application protocol explicitly supports resumable generation with stable offsets and deduplication.

Retry policy

RFC 9110 defines HTTP method semantics, but an LLM POST is not automatically safe to replay. Retry only when all of these are true:

1. No response bytes have crossed the downstream commitment boundary.
2. The request has no unprotected external side effect.
3. The error class is explicitly retryable.
4. The overall deadline leaves useful time for another attempt.
5. The route's attempt cap and fleet-wide retry budget allow it.
6. The selected fallback satisfies the same capability and data-policy contract.

Use jittered backoff and honor applicable Retry-After guidance. A retry budget caps retry traffic relative to healthy traffic; it prevents a failing dependency from turning every original request into several additional requests. Envoy's circuit-breaking guidance separately limits connections, pending requests, active requests, and retries.

Circuit breakers and cancellation

Circuit breakers protect the Gateway and healthy backends from a degraded dependency. Maintain them by backend, endpoint, and sometimes tenant class; a failure in image generation should not necessarily open the chat circuit. Feed passive failure signals into the breaker, but use active probes cautiously because probes consume quota and may not exercise the same path.

Propagate client cancellation upstream. Stop reading, generation, and metering work that is no longer needed, while still writing a terminal usage event. Apply bounded buffers and downstream backpressure so a slow client cannot force unbounded memory growth.

Metering, reservation, and reconciliation

Gateway metering is an operational estimate and allocation ledger, not the final invoice. It can provide fast per-tenant visibility, but provider billing exports remain authoritative for financial reconciliation.

Every terminal attempt should produce an immutable usage record:

{
  "request_id": "req_01",
  "tenant_id": "tenant_red",
  "config_version": "2026-08-09.3",
  "virtual_model": "support-chat",
  "backend_id": "provider_a_chat",
  "attempt": 1,
  "status": "completed",
  "input_units": 1840,
  "output_units": 276,
  "usage_source": "provider_response",
  "price_catalog_version": "catalog_42",
  "estimated_cost": "0.000000",
  "currency": "USD"
}

The numeric values above are illustrative record shapes, not current provider prices. Store money as a decimal or integer minor-unit representation, not binary floating point. Preserve the price-catalog version used for an estimate, and never treat an unknown model, absent usage field, or unmapped billing unit as zero cost. Put it in an exception queue.

Reconciliation compares Gateway records with provider billing data by account, region, model, time window, and available request identifiers. Differences can come from delayed usage, provider-side caching, retries, minimum billing units, rounding, credits, or calls that bypassed the Gateway. The FinOps Open Cost and Usage Specification is useful for normalized allocation and invoice reconciliation, but it does not make a real-time Gateway estimate authoritative.

For a broader comparison of API, self-hosted, and device economics, use AI Inference Cost Economics.

Tenant isolation, cache safety, and secrets

Tenant isolation must cover every stateful surface, not just API authentication. Namespace rate counters, queues, response caches, embeddings, logs, traces, usage ledgers, and administrative queries by the effective tenant and policy scope.

Semantic cache is an authorization decision

A semantic match does not prove that a cached answer is safe or correct for another request. The cache key needs at least:

• tenant and authorization scope;
• normalized model and capability contract;
• system policy and tool definition versions;
• retrieval corpus and data-access version;
• locale and output schema;
• safety policy and cache generation version.

Do not share entries across tenants unless the content is explicitly public and generated under an equivalent contract. Use an offline evaluation set to choose similarity and correctness gates per workload; there is no universal safe threshold or expected hit rate. Collision and poisoning research also shows that semantic proximity can be manipulated, especially when cached outputs can trigger Agent tools. The full lifecycle belongs in Production Semantic Caching.

Secret and egress boundaries

Store provider credentials in a secret manager and deliver short-lived credentials where supported. The data plane should receive only the credentials required for its assigned backends. Restrict egress destinations, validate TLS, rotate keys, and prevent provider headers from being reflected downstream.

The OWASP GenAI LLM Top 10 is a useful threat checklist for prompt injection, sensitive information disclosure, and unbounded consumption. A Gateway can enforce egress and resource policy, but it cannot “solve” prompt injection with a keyword filter.

Telemetry without prompt leakage

Useful Gateway telemetry records decisions and resource behavior without making raw prompts the default payload. Full prompts and model outputs often contain credentials, personal data, retrieved documents, or proprietary source code.

Use three linked signal types:

• metrics for admitted, rejected, queued, active, retried, cancelled, and completed requests plus latency and usage distributions;
• traces for admission, route resolution, upstream attempts, first-byte timing, streaming, and settlement;
• audit and usage events for identity, policy decision, config version, backend, terminal state, and accounting provenance.

Propagate W3C traceparent and tracestate according to the Trace Context Recommendation, while applying its privacy and denial-of-service considerations at trust boundaries. Follow the evolving OpenTelemetry GenAI semantic conventions for interoperable attributes, but pin the convention version in your telemetry contract because the GenAI conventions continue to evolve.

Prefer hashes, lengths, policy labels, model aliases, and sampled redacted excerpts over full content. Put any approved content capture behind separate authorization, retention, encryption, and access-audit controls.

Version, canary, and roll back configuration

Gateway configuration is production code. A malformed alias, overly broad permission, or unsupported fallback can affect every application immediately, so configuration needs the same review and release discipline as binaries.

A safe publication path is:

1. Validate schema, references, capability closure, and security invariants.
2. Sign an immutable snapshot and record its parent version.
3. Load it into test replicas and replay representative requests.
4. Run a shadow evaluation without changing user-visible responses.
5. Canary by tenant, workload, or replica while comparing decision metrics.
6. Promote only if error, rejection, latency, retry, and accounting gates pass.
7. Automatically restore the last known-good snapshot when a gate fails.

Keep control-plane availability separate from data-plane continuity. If the control plane is unavailable, data-plane replicas should continue with a locally verified snapshot for a bounded period. Define whether expired policy fails closed or uses a restricted emergency profile; do not improvise during an incident.

Validate a Gateway policy before rollout

Static validation catches unsafe references before traffic reaches the Gateway. The following standard-library Python program validates a compact JSON policy: backend references, required capabilities, tenant cache namespaces, retry bounds, prompt logging, and budget behavior.

from __future__ import annotations

import argparse
import json
from pathlib import Path
from typing import Any


def require(condition: bool, message: str, errors: list[str]) -> None:
    if not condition:
        errors.append(message)


def string_set(value: Any, field: str, errors: list[str]) -> set[str]:
    if not isinstance(value, list) or any(
        not isinstance(item, str) or not item for item in value
    ):
        errors.append(f"{field} must be a list of non-empty strings")
        return set()
    return set(value)


def validate(policy: dict[str, Any]) -> list[str]:
    errors: list[str] = []
    require(
        isinstance(policy.get("config_version"), str)
        and bool(policy["config_version"]),
        "config_version must be a non-empty string",
        errors,
    )

    backend_rows = policy.get("backends")
    route_rows = policy.get("routes")
    tenant_rows = policy.get("tenants")
    require(isinstance(backend_rows, list), "backends must be a list", errors)
    require(isinstance(route_rows, list), "routes must be a list", errors)
    require(isinstance(tenant_rows, list), "tenants must be a list", errors)
    if not all(isinstance(rows, list) for rows in (
        backend_rows, route_rows, tenant_rows
    )):
        return errors

    backends: dict[str, set[str]] = {}
    for index, row in enumerate(backend_rows):
        if not isinstance(row, dict):
            errors.append(f"backends[{index}] must be an object")
            continue
        backend_id = row.get("id")
        if not isinstance(backend_id, str) or not backend_id:
            errors.append(f"backends[{index}].id must be non-empty")
            continue
        require(
            backend_id not in backends,
            f"duplicate backend id: {backend_id}",
            errors,
        )
        backends[backend_id] = string_set(
            row.get("capabilities"),
            f"backend {backend_id} capabilities",
            errors,
        )
        require(
            isinstance(row.get("secret_ref"), str)
            and bool(row["secret_ref"]),
            f"backend {backend_id} must use secret_ref",
            errors,
        )

    aliases: set[str] = set()
    for index, route in enumerate(route_rows):
        if not isinstance(route, dict):
            errors.append(f"routes[{index}] must be an object")
            continue
        alias = route.get("alias")
        if not isinstance(alias, str) or not alias:
            errors.append(f"routes[{index}].alias must be non-empty")
            continue
        require(alias not in aliases, f"duplicate route alias: {alias}", errors)
        aliases.add(alias)
        required = string_set(
            route.get("required_capabilities"),
            f"route {alias} required_capabilities",
            errors,
        )
        candidates = string_set(
            route.get("candidates"), f"route {alias} candidates", errors
        )
        require(bool(candidates), f"route {alias} has no candidates", errors)
        for backend_id in candidates:
            require(
                backend_id in backends,
                f"route {alias} references unknown backend {backend_id}",
                errors,
            )
            if backend_id in backends:
                missing = required - backends[backend_id]
                require(
                    not missing,
                    f"route {alias} backend {backend_id} lacks "
                    f"{sorted(missing)}",
                    errors,
                )

        retry = route.get("retry")
        if not isinstance(retry, dict):
            errors.append(f"route {alias} retry must be an object")
        else:
            attempts = retry.get("max_attempts")
            require(
                isinstance(attempts, int) and not isinstance(attempts, bool)
                and 1 <= attempts <= 3,
                f"route {alias} max_attempts must be between 1 and 3",
                errors,
            )
            require(
                retry.get("after_downstream_started") is False,
                f"route {alias} must not retry after streaming starts",
                errors,
            )
            ratio = retry.get("budget_ratio")
            require(
                isinstance(ratio, (int, float))
                and not isinstance(ratio, bool)
                and 0 <= ratio <= 1,
                f"route {alias} retry budget_ratio must be in [0, 1]",
                errors,
            )

    for index, tenant in enumerate(tenant_rows):
        if not isinstance(tenant, dict):
            errors.append(f"tenants[{index}] must be an object")
            continue
        tenant_id = tenant.get("id")
        require(
            isinstance(tenant_id, str) and bool(tenant_id),
            f"tenants[{index}].id must be non-empty",
            errors,
        )
        require(
            isinstance(tenant.get("cache_namespace"), str)
            and bool(tenant["cache_namespace"]),
            f"tenant {tenant_id!r} needs an isolated cache_namespace",
            errors,
        )
        require(
            tenant.get("on_budget_exhausted") in {"reject", "manual_approval"},
            f"tenant {tenant_id!r} must reject or require approval "
            "when budget is exhausted",
            errors,
        )

    telemetry = policy.get("telemetry")
    require(isinstance(telemetry, dict), "telemetry must be an object", errors)
    if isinstance(telemetry, dict):
        require(
            telemetry.get("log_prompt_body") is False,
            "telemetry.log_prompt_body must default to false",
            errors,
        )
    return errors


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("policy", type=Path)
    args = parser.parse_args()
    try:
        value = json.loads(args.policy.read_text(encoding="utf-8"))
    except (OSError, json.JSONDecodeError) as error:
        parser.error(str(error))
    if not isinstance(value, dict):
        parser.error("policy root must be an object")
    errors = validate(value)
    if errors:
        for error in errors:
            print(f"ERROR: {error}")
        raise SystemExit(1)
    print(f"valid policy: {args.policy}")


if __name__ == "__main__":
    main()

Run it with python gateway_policy_validator.py gateway-policy.json. A valid file exits with status 0; any invariant violation prints every detected error and exits with status 1. The attempt cap in this example is a local safety invariant, not a universal recommendation. Choose production limits from workload deadlines, side-effect risk, and measured failure behavior.

Production evaluation checklist

Evaluate the Gateway as a critical distributed system, not only as a successful HTTP proxy.

Contract tests

• Replay golden requests for every endpoint, capability, backend, and streaming event.
• Verify unsupported fields fail explicitly instead of disappearing.
• Test cancellation, malformed chunks, missing usage, partial tool calls, and non-JSON errors.
• Confirm aliases never select a backend outside the required region and capability set.

Resilience tests

• Inject connect failures, rate limits, slow headers, mid-stream resets, and control-plane loss.
• Confirm no cross-backend retry occurs after the first downstream byte.
• Measure retry amplification and prove the retry budget closes.
• Saturate concurrency and queues; verify bounded memory and useful rejection reasons.
• Restore the previous configuration snapshot during live traffic.

Accounting and isolation tests

• Reconcile reservations for success, rejection, cancellation, timeout, and ambiguous completion.
• Verify unknown prices and missing usage enter an exception ledger.
• Compare Gateway allocation totals with provider billing exports.
• Attempt cross-tenant cache reads, trace queries, usage queries, and credential access.
• Audit that raw prompts are absent from default logs and traces.

Outcome metrics

Track admitted and accepted workload outcomes separately. An HTTP 200 only means a response traversed the Gateway; it does not prove the model output met the application's quality contract. Gateway SLOs should cover availability, queueing, time to first byte, stream completion, policy correctness, accounting completeness, and isolation. Application teams must separately evaluate output acceptance.

FAQ

How is an LLM Gateway different from a traditional API Gateway?

A traditional API Gateway already provides routing, authentication, TLS, and traffic policy. An LLM Gateway extends that foundation with model aliases, endpoint capability translation, token-aware admission, long-lived streaming, provider usage normalization, and model-specific failure handling. Reuse mature gateway primitives where possible instead of rebuilding them in an LLM SDK wrapper.

Is an OpenAI-compatible endpoint enough for provider portability?

No. It can reduce client integration work for a tested subset, but portability requires contract tests for tools, structured outputs, media, streaming events, errors, usage, and data policy. Maintain a capability matrix, preserve native escape hatches where justified, and reject unsupported combinations.

Can a fallback chain guarantee high availability?

No. A fallback helps only when the alternate backend is healthy, independent enough, contract-compatible, within quota, and reachable before the deadline. Correlated cloud, network, credential, or Gateway failures can defeat the chain. Availability claims require measured SLO evidence from failure injection and production telemetry.

Should the Gateway perform model-quality routing?

The Gateway may enforce an approved alias and candidate set, but prompt classification alone cannot prove that a cheaper backend will produce an acceptable result. Put step-level quality routing, verification, escalation, abstention, and trajectory evaluation in the application or Agent policy described in AI Agent Model Routing.

How should teams adopt an LLM Gateway?

Start with inventory and passive telemetry. Add identity and usage records, then enforce one low-risk quota or route, shadow a versioned policy, canary a bounded tenant, and retain a tested rollback. Migrate provider-specific features only after capability tests prove the translation contract.

Summary

A production LLM Gateway is a narrow, auditable infrastructure boundary. Its data plane enforces identity, capability, quota, streaming, resilience, and metering policy; its control plane versions and safely distributes that policy. The design succeeds when it rejects unsupported contracts, stops retries at the streaming boundary, reconciles rather than guesses cost, isolates every tenant-owned state surface, and can restore a known-good configuration. It fails when a unified URL is mistaken for universal compatibility or a successful HTTP response for an acceptable model outcome.

References and Related Resources

• W3C Trace Context — cross-service trace propagation and security considerations
• RFC 9110: HTTP Semantics — method, intermediary, and retry semantics
• RFC 6585: Additional HTTP Status Codes — 429 Too Many Requests
• Envoy circuit breaking — connection, request, pending, and retry limits
• Envoy AI Gateway v1.0 release documentation — a current implementation example with explicit provider and endpoint coverage
• OpenTelemetry GenAI semantic conventions — evolving GenAI telemetry conventions
• OWASP GenAI LLM Top 10 — security threat categories
• Production Semantic Caching — cache correctness, isolation, and lifecycle
• AI Inference Cost Economics — end-to-end cost and Goodput accounting