What is Prompt Regression Test?
Prompt Regression Test is a repeatable evaluation that compares a candidate prompt or LLM application configuration with a pinned baseline to detect unacceptable loss of previously supported behavior before release.
How It Works
Task, trial, grader, and outcome
A task defines an input, environment, and success criteria. A trial is one attempt because the same task can produce different behavior across runs. A grader measures a specific requirement using code, a calibrated model rubric, or human judgment. For agents, the outcome is the resulting external state, while the transcript records messages and tool calls. Grade the outcome whenever possible: a sentence claiming that a refund succeeded is weaker evidence than the actual order state.
Pinned and paired comparisons
A baseline must pin the prompt version, model identifier, parameters, retrieval corpus, tool definitions, and application code needed to reproduce it. Run baseline and candidate against the same task IDs and comparable environments; otherwise a dataset or dependency change can be mistaken for a prompt regression. Preserve per-task paired results rather than comparing only two global averages, because a small gain on frequent easy cases can hide a severe loss in a safety or high-value slice.
Graders and uncertainty
Use deterministic graders for parsing, schemas, required citations, tool arguments, policy invariants, and end-state checks. Use rubric-based or pairwise model graders for qualities such as relevance or tone, and calibrate them against expert labels. Exact string equality is usually inappropriate for open-ended output. When model or grader variance can change verdicts, repeat trials and report intervals or another justified uncertainty estimate rather than treating a one-run score difference as fact.
Release gates and production feedback
Define gates before running the candidate: zero tolerance for security or authorization failures, per-slice floors for critical intents, and bounded changes for aggregate quality, latency, tokens, and cost. Inspect newly failing tasks instead of trusting one blended score. Add confirmed incidents and representative production traces to the suite after privacy review, but keep a held-out set to detect overfitting. Regression tests are pre-release evidence; drift monitoring and controlled online experiments remain necessary after deployment.
Key Characteristics
- Compares a candidate against a pinned baseline on the same task identities
- Uses repeated trials when model or grader variance can alter the verdict
- Combines deterministic, model-based, and human graders by requirement
- Tracks aggregate results and critical slices instead of one blended score
- Includes real failures, representative traffic, edge cases, and adversarial inputs
- Gates releases while leaving post-release drift detection as a separate control
Common Use Cases
- Comparing a rewritten system prompt with the current production baseline
- Checking schema, refusal, citation, and tool-policy invariants before release
- Evaluating a model or decoding-parameter migration on existing user intents
- Replaying confirmed production failures after a corrective change
- Blocking an agent release when critical outcome or cost thresholds regress
Example
Loading code...Frequently Asked Questions
What changes should trigger prompt regression tests?
Run them for prompt or template edits, model and parameter changes, retrieval or chunking updates, tool schema and permission changes, grader revisions, and orchestration changes. The test scope should match the affected behavior; a model migration may require the full suite, while a narrow formatting edit may start with relevant slices.
Why should baseline and candidate runs be paired by task?
Pairing controls for task difficulty and exposes exactly which capabilities changed. Comparing unrelated sample sets or only global averages can turn dataset composition into a false improvement and can hide regressions in a critical intent behind gains elsewhere.
Should a regression test compare exact LLM output text?
Usually not for open-ended tasks because several responses may be valid. Use exact checks for deterministic contracts such as parseability, enum membership, required fields, tool permissions, and end state. Use calibrated rubrics, pairwise judgments, or human review for semantic quality while retaining concrete failure examples.
How should non-deterministic results be handled?
Repeat trials under pinned settings, retain per-task outcomes, and report an uncertainty interval or another preselected statistical test. Do not rerun only failures until they pass. A release rule should distinguish tolerated variance from material regression and use stricter zero-tolerance gates for security or authorization invariants.
Should production examples be added to the regression suite?
Yes, after privacy review, deduplication, and labeling. Confirmed incidents and representative traffic keep the suite relevant, while adversarial and boundary cases protect costly failure modes. Maintain a held-out set and review the distribution periodically so repeated prompt tuning does not overfit a familiar golden dataset.