Prompt CI/CD: Versioning, Evals, Release Gates, Rollback

TL;DR

Prompt CI/CD is a release discipline for changes to LLM application behavior. The deployable unit is not a prompt string; it is an immutable bundle of prompts, model revisions, parameters, tools, retrieval dependencies, schemas, datasets, evaluators, and policy. A reliable pipeline verifies deterministic contracts, compares repeated offline runs, calibrates judge-assisted metrics, applies risk-specific gates, then uses guarded online experiments with monitoring and complete rollback.

Table of Contents

1. What Prompt CI/CD Actually Releases
2. Build an Immutable Release Bundle
3. Design the Offline Evaluation Gate
4. Use LLM Judges Without Treating Them as Ground Truth
5. Implement a Paired Regression Gate
6. Separate Offline Evals from Online Experiments
7. Secure the CI Trust Boundary
8. Monitor and Roll Back the Complete Bundle
9. Failure Modes and Review Checklist
10. FAQ
11. References

Key Takeaways

• Release a behavior bundle, not a text file. Prompt text cannot identify behavior without the model, tools, retrieval state, schema, and runtime policy.
• Use deterministic checks before probabilistic scoring. Schema validity, forbidden tool calls, citation requirements, and authorization rules should not depend on a judge model.
• Compare matched examples and repeated runs. Aggregate means can hide regressions in safety, language, tenant, or task slices.
• Calibrate every automated judge. A paper's human agreement result is not a reusable accuracy guarantee for another rubric or dataset.
• Keep offline and online evidence separate. Offline regression tests reduce pre-release risk; online experiments estimate real-traffic impact.
• Rollback by immutable release identity. Restoring only the prompt can preserve the model, index, schema, or tool change that caused the incident.

What Prompt CI/CD Actually Releases

Prompt CI/CD continuously integrates, evaluates, releases, observes, and rolls back changes that can alter an LLM application's behavior. It extends Prompt Versioning beyond text history and connects each release candidate to evidence.

The common phrase "prompts are code" is useful but incomplete. A source diff explains what a person edited. It does not prove which runtime behavior was evaluated or deployed. The same prompt can produce materially different results after any of these changes:

The release decision therefore needs two identities:

1. Source identity records authorship, review, and history, usually through Git.
2. Runtime identity records the exact bundle that produced and evaluated behavior.

Git is an excellent system of record for source changes. It is not, by itself, an experiment registry, artifact store, deployment controller, or proof of behavioral reproducibility.

Build an Immutable Release Bundle

An immutable release manifest should identify every behavior-affecting dependency by a content digest, immutable revision, or artifact ID. Mutable aliases such as latest, a provider's floating model name, or an unversioned vector index are convenient selectors, not release identities.

{
  "releaseId": "support-rag/sha256:4e89c6...",
  "sourceCommit": "9f5cb7...",
  "prompt": {
    "artifact": "prompts/support-answer.json",
    "sha256": "c8f4a2..."
  },
  "model": {
    "provider": "provider-a",
    "revision": "immutable-model-revision",
    "parameters": {
      "temperature": 0.2,
      "maxOutputTokens": 900
    }
  },
  "tools": {
    "schemaSha256": "2a70b1...",
    "policySha256": "79945d..."
  },
  "retrieval": {
    "corpusSnapshot": "support-docs-184",
    "indexBuild": "index-2026-08-09-03"
  },
  "output": {
    "schemaSha256": "b5cb3a...",
    "parserRevision": "parser/7d34fe..."
  },
  "evaluation": {
    "datasetRevision": "support-eval/31",
    "runnerRevision": "eval-runner/8c18a0...",
    "judgeRevision": "judge-rubric/12"
  }
}

The manifest should be canonicalized and hashed after validation. Store the immutable manifest in an artifact registry and let a separately controlled deployment pointer reference its digest. A rollback changes that pointer to a previously approved digest; it does not construct a new approximation of the old release.

During review, use Code Diff to inspect prompt, policy, and manifest changes together. Use JSON Formatter to inspect a generated manifest or gate report before signing the artifact; neither tool replaces schema validation or digest verification.

Inventory the change impact

Before running expensive evals, classify what changed:

This impact inventory prevents both extremes: rerunning every test for a documentation-only change, or approving a model migration with a narrow prompt-only suite.

Design the Offline Evaluation Gate

An offline gate should combine deterministic invariants, task metrics, risk slices, repeated sampling, and explicit release policy. OpenAI's evaluation best practices similarly recommend task-specific datasets, production-like distributions, continuous evaluation, and human calibration rather than generic "vibe" checks.

Layer 1: deterministic contracts

Use normal programs for claims that have a machine-checkable answer:

• output parses and validates against the expected schema;
• required fields and citations are present;
• tool names and arguments match an allowlist;
• authorization is checked outside the model;
• latency, token, and call budgets remain within policy;
• secrets and personal data do not appear in logs;
• known prompt-injection fixtures cannot bypass runtime controls.

These tests should fail closed. A judge must not overrule a missing authorization check or malformed transaction payload.

Layer 2: task-specific regression cases

Build the dataset from production-like inputs, incident cases, domain-expert cases, adversarial cases, and known edge conditions. Keep a held-out set for release decisions, and label dimensions that matter:

task: refund_explanation
locale: en-US
risk: financial
tenant_policy: standard
input_source: production_sample
expected_contract: cited_answer_without_mutation

There is no universal correct count for a golden set. A small set can completely cover a finite schema invariant but cannot estimate a small product effect. A large random sample can estimate an average while missing a rare, high-impact safety path. Dataset sufficiency depends on coverage, baseline rates or variance, minimum detectable effect, desired statistical power, allocation, clustering, repeated model calls, and the risk slices that must be protected.

Layer 3: repeated stochastic runs

One output per case can confuse random variation with a behavioral change. For nondeterministic paths:

1. run baseline and candidate on the same case IDs;
2. repeat each condition under a declared sampling policy;
3. preserve raw outputs, latency, token usage, tool traces, and evaluator decisions;
4. compare paired differences, not unrelated aggregate means;
5. report confidence intervals and per-slice results;
6. require manual adjudication when evidence is inconclusive for a consequential change.

The release policy should define what evidence is required before the results exist. Selecting a threshold after seeing the experiment converts a gate into a negotiation.

Use LLM Judges Without Treating Them as Ground Truth

LLM-as-a-Judge can scale rubric-based comparison of open-ended outputs, but its score is a measurement produced by another model. The MT-Bench paper reports strong agreement in its own models, questions, prompts, and annotation setup; it also documents position, verbosity, self-enhancement, and reasoning limitations. That result is not a universal judge accuracy.

A defensible judge workflow includes:

1. Define an observable rubric. Replace "good answer" with separate criteria such as factual support, instruction adherence, completeness, harmful action, and citation validity.
2. Prefer comparison where appropriate. Pairwise judgments can reveal small relative changes, while reference-guided grading works for tasks with a verified answer.
3. Blind and randomize. Hide version identity and randomize answer order; repeat swapped-order comparisons to measure position sensitivity.
4. Pin judge identity. Record model revision, prompt, rubric, decoding settings, parser, and retry policy.
5. Calibrate against humans. Use a blinded, representative human-labeled sample and report agreement and error by slice.
6. Adjudicate disagreements. Route judge-human disagreement, low-evidence outputs, and high-risk cases to qualified reviewers.
7. Recalibrate after change. A new judge model, rubric, dataset distribution, or product task invalidates the previous calibration assumption.

Do not use the judge's self-reported confidence as a probability that its verdict is correct. If a calibrated probability is required, estimate it from held-out labeled data and monitor calibration drift.

Implement a Paired Regression Gate

The following dependency-free Python example validates an immutable manifest, joins baseline and candidate results by case and repeat, calculates paired bootstrap confidence intervals, and evaluates policy by metric and slice. The thresholds are inputs owned by the application team; the code deliberately contains no universal five-percent rule.

from __future__ import annotations

import hashlib
import json
import random
from collections import defaultdict
from pathlib import Path
from statistics import fmean
from typing import Any


REQUIRED_MANIFEST_PATHS = (
    ("prompt", "sha256"),
    ("model", "revision"),
    ("tools", "schemaSha256"),
    ("tools", "policySha256"),
    ("retrieval", "indexBuild"),
    ("output", "schemaSha256"),
    ("evaluation", "datasetRevision"),
    ("evaluation", "runnerRevision"),
)


def nested_value(document: dict[str, Any], path: tuple[str, ...]) -> Any:
    value: Any = document
    for key in path:
        if not isinstance(value, dict) or key not in value:
            raise ValueError(f"missing manifest field: {'.'.join(path)}")
        value = value[key]
    return value


def release_digest(manifest: dict[str, Any]) -> str:
    for path in REQUIRED_MANIFEST_PATHS:
        nested_value(manifest, path)
    payload = json.dumps(
        manifest, sort_keys=True, separators=(",", ":"), ensure_ascii=False
    ).encode("utf-8")
    return "sha256:" + hashlib.sha256(payload).hexdigest()


def paired_differences(
    baseline: list[dict[str, Any]],
    candidate: list[dict[str, Any]],
    metric: str,
    slice_name: str,
) -> list[float]:
    def select(rows: list[dict[str, Any]]) -> dict[tuple[str, int], float]:
        selected = {}
        for row in rows:
            if row["slice"] == slice_name:
                selected[(row["caseId"], row["repeat"])] = float(row[metric])
        return selected

    left, right = select(baseline), select(candidate)
    if left.keys() != right.keys() or not left:
        raise ValueError(f"unpaired or empty results for {metric}/{slice_name}")
    return [right[key] - left[key] for key in sorted(left)]


def bootstrap_interval(
    values: list[float], confidence: float, samples: int, seed: int
) -> tuple[float, float]:
    rng = random.Random(seed)
    estimates = sorted(
        fmean(rng.choice(values) for _ in values) for _ in range(samples)
    )
    tail = (1.0 - confidence) / 2.0
    low = estimates[int(tail * (samples - 1))]
    high = estimates[int((1.0 - tail) * (samples - 1))]
    return low, high


def evaluate_gate(
    baseline: list[dict[str, Any]],
    candidate: list[dict[str, Any]],
    policy: dict[str, Any],
) -> dict[str, Any]:
    checks = []
    for rule in policy["rules"]:
        diffs = paired_differences(
            baseline, candidate, rule["metric"], rule["slice"]
        )
        low, high = bootstrap_interval(
            diffs,
            confidence=policy["confidence"],
            samples=policy["bootstrapSamples"],
            seed=policy["seed"],
        )
        # "higher" metrics fail when the lower bound misses the allowed delta.
        # "lower" metrics fail when the upper bound exceeds the allowed delta.
        passed = (
            low >= rule["minimumDelta"]
            if rule["direction"] == "higher"
            else high <= rule["maximumDelta"]
        )
        checks.append(
            {
                "metric": rule["metric"],
                "slice": rule["slice"],
                "pairs": len(diffs),
                "meanDelta": fmean(diffs),
                "interval": [low, high],
                "passed": passed,
            }
        )
    return {"passed": all(item["passed"] for item in checks), "checks": checks}


def main() -> None:
    manifest = json.loads(Path("release-manifest.json").read_text())
    baseline = json.loads(Path("baseline-results.json").read_text())
    candidate = json.loads(Path("candidate-results.json").read_text())
    policy = json.loads(Path("release-policy.json").read_text())
    report = {
        "releaseDigest": release_digest(manifest),
        **evaluate_gate(baseline, candidate, policy),
    }
    Path("gate-report.json").write_text(
        json.dumps(report, indent=2, ensure_ascii=False) + "\n"
    )
    if not report["passed"]:
        raise SystemExit("release gate failed; inspect gate-report.json")


if __name__ == "__main__":
    main()

An application-owned policy can protect average quality and a high-risk slice independently:

{
  "confidence": 0.95,
  "bootstrapSamples": 10000,
  "seed": 20260809,
  "rules": [
    {
      "metric": "taskScore",
      "slice": "all",
      "direction": "higher",
      "minimumDelta": -0.01
    },
    {
      "metric": "unsafeActionRate",
      "slice": "financial",
      "direction": "lower",
      "maximumDelta": 0.0
    }
  ]
}

Bootstrap intervals do not repair a biased dataset, invalid judge, repeated-user dependence, or underpowered design. Treat the report as one evidence artifact, not proof that the release is universally safe.

Separate Offline Evals from Online Experiments

Offline evals and online experiments answer different questions. The first asks whether a candidate satisfies known contracts and improves or preserves measured behavior on a controlled dataset. The second asks what happens under real users, traffic, latency, and feedback loops.

Predeclare the experiment contract

Before exposure, record:

• experimental unit and stable assignment key;
• control and treatment release digests;
• eligible population and exclusions;
• one primary metric and its minimum meaningful effect;
• safety, quality, latency, cost, and complaint guardrails;
• sample-size or sequential design assumptions;
• exposure duration and stopping rules;
• sample-ratio mismatch checks;
• multiple-comparison handling;
• rollback owner and kill switch.

p < 0.05 alone is not a rollout rule. A decision also needs effect size, uncertainty interval, practical significance, intact randomization, acceptable guardrails, and evidence that novelty, seasonality, carryover, or repeated-user effects do not explain the result. Safety invariants are constraints, not metrics to trade for engagement.

High-impact safety or authorization changes should first be evaluated in isolated environments, simulations, red-team exercises, or shadow traffic. Do not intentionally expose users to a candidate merely to discover whether it violates a known safety contract.

Secure the CI Trust Boundary

A Prompt CI job processes untrusted material: pull-request code, prompt text, retrieved fixtures, model outputs, comments, commit messages, and generated artifacts. Model credentials and deployment permissions must not share that trust boundary.

GitHub's secure use reference recommends minimum token permissions, warns against privileged triggers that check out untrusted pull-request code, and states that pinning an Action to a full commit SHA is the immutable form of reference.

A safe pipeline separates jobs:

1. Unprivileged pull-request job
   • read-only repository permission;
   • no model, production, or deployment secrets;
   • schema, manifest, static fixture, and deterministic tests;
   • uploads a narrowly validated result artifact.
2. Approved evaluation job
   • runs trusted evaluator code from the protected base revision;
   • reads candidate artifacts as data, never executes them;
   • uses short-lived, least-privilege credentials;
   • writes an attestable report, not repository content.
3. Deployment job
   • requires protected-environment approval for consequential releases;
   • accepts only approved release digests;
   • cannot be modified or approved by the same automation it deploys.

name: prompt-contracts
on:
  pull_request:
    paths:
      - "prompts/**"
      - "evals/**"
      - "release-policy.json"

permissions:
  contents: read

jobs:
  deterministic-contracts:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - name: Check trusted repository workspace
        run: |
          test -f scripts/validate_release.py
          python scripts/validate_release.py --no-network

This minimal example intentionally uses no secrets and grants no write permission. If checkout, setup, upload, or third-party Actions are added, pin each reference to a verified full commit SHA. Never interpolate prompt text or pull-request metadata directly into generated shell code or $GITHUB_OUTPUT; pass untrusted values as files or quoted environment data and validate their size and schema.

Monitor and Roll Back the Complete Bundle

A release that passes CI can still fail under distribution shift, provider changes, retrieval freshness, tool outages, or unexpected user behavior. Monitoring must preserve the release digest and the evidence needed to distinguish these causes.

Track at least:

• release and experiment assignment identity;
• schema and tool-call validity;
• task success and grounded-answer metrics;
• safety and authorization violations;
• latency, token use, retries, and cost;
• fallback and refusal rates;
• user complaints and qualified human review;
• metrics by locale, tenant, task, and risk slice;
• model, retrieval, and tool dependency health.

Rollback is a transaction across dependencies. Before promotion, verify that the prior bundle remains deployable, its index and model revision still exist, downstream schemas remain compatible, migrations are reversible or forward-compatible, and routing caches can converge. A rollback drill is stronger evidence than a document claiming rollback support.

For broader production telemetry, see the AI agent observability guide. For runtime routing and fallback isolation, see the LLM gateway architecture guide. Prompt injection defenses still belong in the runtime trust model; regression fixtures complement, but do not replace, the controls in the prompt injection defense guide.

Failure Modes and Review Checklist

Before approving a release, reviewers should be able to answer:

• Which exact bundle was tested, and can it be reconstructed?
• Which deterministic contracts must never regress?
• Does the dataset reflect production tasks and high-impact slices?
• Are baseline and candidate results paired under the same conditions?
• How was any judge calibrated, versioned, and checked for bias?
• Were thresholds and online stopping rules declared before results?
• Can the CI path expose secrets or execute untrusted artifacts?
• What observation triggers rollback, and has full rollback been tested?

FAQ

What must be versioned in Prompt CI/CD?

Version every dependency that can change observable behavior: prompt and template, model revision, decoding settings, tools and policy, retrieval snapshot, output schema and parser, dataset, evaluator, and routing policy. Git records source history; a canonical manifest digest identifies the runtime release.

How many examples does a prompt regression test need?

No fixed number works across tasks. Deterministic contract coverage, rare safety cases, average quality estimation, and online product effects have different data requirements. Declare the target effect, uncertainty, power, repeats, slices, and stopping rule, then justify coverage for high-impact cases separately.

Is LLM-as-a-Judge reliable enough for a release gate?

It can contribute evidence after calibration. Use explicit rubrics, blinded and randomized comparisons, held-out human labels, slice-level disagreement analysis, pinned judge revisions, and human adjudication. Do not treat a judge's score or self-confidence as ground truth.

Are offline evals a substitute for A/B tests?

No. Offline evals catch known regressions without exposing users; online experiments estimate impact under live traffic. A candidate must first satisfy deterministic safety and compatibility gates. An online win cannot compensate for a failed authorization or safety invariant.

How should a team control Prompt CI/CD cost?

Measure cost from the actual suite: case count, repeats, input and output tokens, judge calls, retries, storage, and human review. Run cheap deterministic checks first, select affected suites from the change inventory, cache only immutable inputs and outputs, and reserve expensive evaluation for candidates that survive early gates. Avoid publishing a universal per-run cost.

References

• OpenAI: Evaluation best practices
• GitHub: Secure use reference for GitHub Actions
• Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• NIST AI 600-1: Generative Artificial Intelligence Profile
• Prompt CI/CD glossary
• Prompt regression test glossary
• Golden dataset glossary