Constrained Decoding: Schema-Guided LLM Output

TL;DR

Constrained decoding enforces an output language while an LLM generates tokens. It can make output conform to a supported JSON Schema, grammar, regex, or list of choices, but it does not make the values true or the action safe. Production systems need a complete contract: constraint compilation, explicit refusal and truncation handling, application validation, observability, and a fail-closed policy for high-impact workflows.

What constrained decoding changes

Constrained decoding changes where format enforcement happens. Prompt-only formatting asks the model to cooperate. Post-generation validation rejects or repairs an answer after the model has already produced it. Constrained decoding prevents invalid continuations from being sampled in the first place.

At generation step t, a normal decoder samples from the model distribution:

P(token_t | prompt, token_1 ... token_t-1)

A constrained decoder intersects that distribution with tokens allowed by the current grammar state:

allowed = grammar.valid_tokens(generated_prefix)
masked_logits[token not in allowed] = -infinity
token_t = sample(masked_logits)

The important phrase is current grammar state. The engine does not merely check whether the final string parses. It tracks whether the current prefix can still become a valid member of the requested language.

Prompting, JSON mode, schemas, and validation

These mechanisms solve different parts of the output problem and should not be treated as interchangeable.

Structured output is the overall application goal. JSON mode and constrained decoding are possible mechanisms. A JSON Schema is a contract description, while a provider's decoding backend decides which subset it can enforce.

OpenAI's Structured Outputs documentation describes strict JSON Schema output for supported models. NVIDIA NIM documents guided JSON, regular expressions, context-free grammars, and fixed choices, while also noting backend-specific support and fallback behavior. Those differences are why “supports structured output” is not a sufficient compatibility statement.

How a schema becomes token rules

Schema-guided generation typically passes through four layers:

1. Normalize the constraint. Resolve references, validate supported keywords, and reject ambiguous or unsupported constructs.
2. Compile a recognizer. Convert the schema or grammar into a finite-state representation, pushdown automaton, or optimized matcher.
3. Map grammar state to tokenizer tokens. A token can represent one character, several characters, or part of a Unicode sequence. The engine must test token text against valid continuations.
4. Mask and advance. At every step, block invalid tokens, sample a valid token, and update the recognizer state.

This creates practical edge cases:

• a schema can be valid JSON Schema but outside the provider's supported subset;
• a regular expression can compile under one backend and fail under another;
• tokenizer vocabulary can make apparently simple constraints expensive;
• a valid empty object may satisfy JSON mode but violate the application's expected fields;
• streaming consumers can receive incomplete prefixes that are not yet parseable documents;
• safety refusal or token limits can end the request without a schema instance.

Designing a production output contract

A production contract must define behavior outside the happy path. Start with a narrow schema and make uncertain data explicit instead of forcing the model to invent a value.

{
  "type": "object",
  "properties": {
    "ticket_id": { "type": ["string", "null"] },
    "priority": {
      "type": "string",
      "enum": ["low", "medium", "high", "unknown"]
    },
    "evidence_ids": {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "required": ["ticket_id", "priority", "evidence_ids"],
  "additionalProperties": false
}

The executor should then apply deterministic checks:

type Ticket = {
  ticket_id: string | null;
  priority: "low" | "medium" | "high" | "unknown";
  evidence_ids: string[];
};

function validateForEscalation(
  ticket: Ticket,
  visibleEvidenceIds: Set<string>,
): void {
  if (ticket.priority === "high" && ticket.evidence_ids.length === 0) {
    throw new Error("high_priority_requires_evidence");
  }
  for (const id of ticket.evidence_ids) {
    if (!visibleEvidenceIds.has(id)) {
      throw new Error("unavailable_evidence");
    }
  }
}

The decoder guarantees neither rule. It does not know which evidence the caller is authorized to see, and a Schema cannot prove that a high-priority classification has support.

Latency and throughput trade-offs

Constrained decoding can remove repair retries while still adding work to the serving path. Measure both sides.

Do not assume a more restrictive grammar is always faster. Some backends optimize common JSON Schema paths, while complex regex or grammar combinations can fall back to a slower implementation. Benchmark the exact model, tokenizer, backend, schema family, streaming mode, and concurrency used in production.

Failure modes and safe fallback

Treat these outcomes as different states:

A fallback to prompt-only JSON may be acceptable for a low-risk draft shown to a human. It is not acceptable when the result triggers a payment, permission change, database write, or external message. In those paths, inability to enforce the contract must stop execution.

Testing and observability

Test the constraint layer separately from model quality.

Conformance suite

• valid minimal, typical, and maximal objects;
• Unicode, escaped text, empty arrays, and long strings;
• unsupported keywords and recursive schemas;
• refusal, cancellation, timeout, and truncation;
• streaming assembly and partial consumer disconnect;
• backend upgrade and tokenizer change.

Semantic suite

• wrong but well-typed values;
• mutually inconsistent fields;
• invented identifiers;
• unauthorized resources;
• missing evidence;
• adversarial text asking the model to alter the schema.

Record schema version and digest, backend, model revision, compilation result, first-token latency, output-token latency, finish reason, refusal state, validation result, and fallback path. Avoid recording sensitive raw payloads by default.

Decision checklist

Use constrained decoding when downstream code needs a narrow machine-readable contract and the deployed backend supports it reliably. Prefer simpler controls when the final output is prose for a human.

Before release:

1. Confirm the provider's supported schema or grammar subset.
2. Version and review the constraint as production code.
3. Define refusal, truncation, and unsupported-schema outcomes.
4. Validate domain rules and authorization after parsing.
5. Benchmark cold and warm schemas under production concurrency.
6. Add conformance canaries for backend upgrades.
7. Prevent silent fallback on consequential actions.

For tool execution, pair this contract with the host-side authorization model described in the function calling guide. For general JSON validation, the JSON Schema guide covers application-side validation in more depth.

FAQ

Can constrained decoding force the model to answer?

No. Safety refusal, unavailable information, cancellation, and token limits remain possible. The response protocol must represent those states explicitly rather than forcing them into a normal data object.

Is a grammar better than a JSON Schema?

It depends on the language. JSON Schema is convenient for typed application objects. A context-free grammar is better for a DSL or syntax that is not JSON. Regex is suitable only for bounded regular languages and becomes hard to maintain for nested structures.

Should schemas be generated dynamically?

Only within strict limits. Dynamic schemas expand the compilation cache, attack surface, and compatibility matrix. Prefer reviewed templates with bounded parameters, size limits, and canonicalization.

Can constrained decoding be combined with function calling?

Yes. Providers often use schema-constrained generation to produce tool arguments. The model still proposes an action; trusted code must validate and authorize the exact call before execution.

How do I know whether a provider silently downgraded the constraint?

Run negative canaries that the unrestricted model would likely violate, record backend and feature flags, and validate every response independently. A provider success status is not proof that the requested constraint was applied.

Sources

• OpenAI Structured Outputs documentation, accessed 2026-07-28.
• NVIDIA NIM structured generation documentation, accessed 2026-07-28.
• NVIDIA Triton guided decoding documentation, accessed 2026-07-28.