What is Human-in-the-Loop?
Human-in-the-Loop is a control pattern in which an AI system pauses at defined decision points so an authenticated, authorized person can review, approve, reject, edit, escalate, or halt proposed work before execution continues.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Human-in-the-Loop (HITL) is not a prompt telling a model to ask permission and not a vague claim that someone can inspect logs later. It is an enforced control path between a proposed decision and its consequence. The runtime or downstream service determines when review is mandatory; the model cannot grant itself an exception. Human-on-the-Loop monitoring, post-event audit, and dataset labeling are useful human controls, but they do not provide the same pre-execution gate.
Separate interrupt, approval, and authorization. An interrupt persists the run and exposes a pending item. An approval records a person's decision about one concrete proposal. Authorization checks whether the actor, approver, tenant, resource, operation, and current policy allow execution. An approval is evidence for authorization, not a replacement for it; the downstream service still enforces ownership and least privilege.
A durable Approval Envelope should bind approval_id, run_id, call_id, actor, authenticated approver, reviewer role, tenant, resource, exact tool and schema version, argument digest, policy version, decision, rationale or edit, issue time, expiry, and single-use state. The UI should display the exact target, salient argument diff, expected external effect, evidence, uncertainty, reversibility, and alternatives while withholding secrets and hidden chain-of-thought. A generic "approve this agent" button is too broad.
Review responses include approve, reject, edit, request clarification, escalate, and cancel. Editing an argument creates a new proposal and digest; it must pass schema, policy, and any required approval again. Reject should produce a terminal or explicitly routed state rather than invite the model to rephrase the same request indefinitely. Expired, duplicate, unauthorized, mismatched, or already consumed decisions fail closed.
Long-running review requires durable checkpoints and a pending-approval store independent of one process or browser session. Resume the same run from compatible state and revalidate the proposal against current resource state, policy revision, tool schema, approver authority, and expiry immediately before execution. Framework replay may restart code before an interrupt, so any earlier side effect must be absent or idempotent. A dispatch timeout after approval is an Effect Journal problem, not a reason to request approval and execute again blindly.
Select gates by action-level risk, not by agent label or an uncalibrated model confidence. Consider impact, reversibility, data sensitivity, delegated authority, uncertainty, novelty, and policy. Low-risk reads may use deterministic controls; high-consequence writes often need approval or separation of duties. Human review cannot repair excessive permissions, unsafe tools, prompt injection, or missing downstream authorization, so combine it with narrow capabilities, isolation, budgets, validation, and complete mediation.
Measure more than approval rate. Test bypass attempts, cross-tenant reviewers, changed arguments, stale policy and schema, expiry, replay, concurrent decisions, cancellation, reviewer unavailability, and crash before or after execution. Track gate coverage, false allow and false block rates, disagreement, edit and escalation rates, queue age, P50/P95 decision latency, abandoned items, override outcomes, incidents, and reviewer load. Reduce repetitive low-value prompts with deterministic policy, but never automate a high-risk class solely because reviewers usually click approve.
Key Characteristics
- Enforced pause before a defined consequence, independent of model cooperation
- Authenticated reviewer and explicit separation between interrupt, approval, and authorization
- Proposal-bound decision covering actor, resource, arguments, schema, policy, expiry, and single use
- Durable pending state that can survive process loss and resume the same compatible run
- Approve, reject, edit, clarify, escalate, cancel, and emergency-halt paths
- Coverage, decision quality, latency, fatigue, bypass, and incident metrics
Common Use Cases
- Approving a refund, payment, account change, deployment, merge, deletion, or external message
- Correcting extracted contract or invoice fields before they enter a system of record
- Escalating novel, ambiguous, high-impact, or policy-sensitive cases to a domain expert
- Pausing a long-running agent and resuming it after an asynchronous review
- Separating the requester's authority from an independent reviewer for privileged operations
- Capturing reviewed examples for evaluation without treating approval as a training label by default
Example
Loading code...Frequently Asked Questions
Is Human-in-the-Loop the same as asking the model for confirmation?
No. A prompt can be ignored or manipulated. A real gate is enforced by trusted runtime or downstream code and prevents the sensitive operation from executing until a valid decision is supplied.
Does human approval replace authorization?
No. Approval records a review decision. Authorization must still verify the actor, approver authority, tenant, resource ownership, current policy, exact arguments, and least-privilege credential at execution time.
What happens when a reviewer edits a tool call?
The edit creates a new proposal and argument digest. Validate its schema and business rules, rerun policy, and obtain fresh approval when the changed operation still requires it. Never reuse approval for different arguments.
How should delayed approvals be resumed?
Persist the run and pending Approval Envelope, authenticate the reviewer, then resume the same compatible run. Before executing, recheck expiry, single-use status, policy and schema versions, resource state, and approver authority.
How is a Human-in-the-Loop system evaluated?
Test bypass, stale or replayed approvals, argument changes, unauthorized reviewers, timeouts, concurrent decisions, crashes, and cancellation. Measure coverage, false allows and blocks, latency, edits, escalations, abandonment, fatigue, and incidents.