Agentic RAG: Control Loops, Evidence, and Evaluation

Direct Answer

Agentic RAG is a retrieval architecture in which an AI agent decides whether, where, and how often to retrieve instead of executing one fixed search. The production unit is not an unconstrained “thinking” loop. It is a bounded evidence-control loop with authorized retrieval tools, typed results, provenance, iteration and token budgets, explicit stop reasons, and a final policy that either synthesizes from sufficient evidence, asks for clarification, or abstains. Agentic behavior is valuable only when it improves accepted answers or goodput over a strong fixed-RAG baseline.

Key Takeaways

• Retrieval becomes a tool under policy control; it does not become trusted merely because an LLM selected it.
• Self-RAG, CRAG, and Adaptive-RAG are distinct research methods, not interchangeable labels for any loop with a grader.
• The loop needs deterministic limits for steps, tools, tokens, time, evidence age, and side effects.
• An LLM score is a noisy signal. It cannot grant authorization, prove factual support, or sanitize prompt injection.
• Evaluate route selection, retrieval, evidence, answer, abstention, security, and operations separately before rollout.

What Makes RAG Agentic?

RAG becomes agentic when retrieval decisions move from a fixed application graph into a runtime control policy. A standard pipeline can already use hybrid search, reranking, metadata filters, and citations. Agentic RAG adds conditional decisions such as:

• skip retrieval, search one source, or query several authorized sources;
• decompose a multi-part question into evidence tasks;
• reformulate a failed query without changing the user's goal;
• inspect gaps or conflicts in retrieved evidence;
• stop, clarify, abstain, or run another bounded step.

This definition does not require visible chain-of-thought, multiple agents, or a particular framework. It does require state and conditional control. The Agentic RAG survey describes a broader taxonomy across agent cardinality, control structure, autonomy, and knowledge representation. Therefore, “routing, multi-step, corrective, and adaptive” are useful examples, not four canonical and exhaustive production modes.

The loop is part of an agentic workflow, but retrieval remains a read operation with its own contracts. Business actions such as updating an order or issuing a refund should use separate tools, authorization, confirmation, idempotency, and audit policies.

Research Patterns Have Different Contracts

The most cited adaptive retrieval methods solve different problems. Treating them as synonyms creates incorrect implementations and misleading claims.

Self-RAG specifically trains a model to generate reflection tokens. Calling several ordinary model APIs to critique an answer may be self-reflective orchestration, but it is not the same trained method.

Corrective RAG evaluates retrieval quality and selects different corrective actions. The evaluator is still fallible, and an external web fallback changes trust, authorization, freshness, and citation requirements.

Adaptive-RAG learns a query-complexity classifier from task outcomes and dataset signals. Its no-retrieval, single-step, and iterative routes are an experimental design, not universal query classes.

Paper results belong to each paper's models, datasets, retrievers, and evaluation protocol. Use them to understand mechanisms, then establish local evidence.

Design Retrieval Tools as Security Boundaries

A retrieval tool is an application-owned interface, not a raw database handle or arbitrary URL fetcher. Its schema should make the authorization and evidence contract explicit.

{
  "name": "search_support_runbooks",
  "purpose": "Find approved operational runbooks for the caller's tenant",
  "input": {
    "query": "string",
    "tenant_id": "server-derived",
    "product": "allowlisted enum",
    "top_k": "integer between 1 and 8"
  },
  "output": {
    "evidence": [
      {
        "source_id": "immutable document ID",
        "revision": "immutable revision",
        "title": "display title",
        "excerpt": "bounded text",
        "retrieved_at": "timestamp",
        "acl_scope": "authorization scope"
      }
    ]
  },
  "limits": {
    "timeout_ms": 1500,
    "max_result_bytes": 40000
  }
}

Microsoft's current agentic RAG architecture guidance similarly recommends specific tool descriptions, typed parameters, return schemas, metadata, and filters. Those practices help selection, but the service must still derive identity and tenant scope outside the model.

Authorization Must Precede Retrieval

The agent must never invent tenant_id, access filters, database names, or credentials. The application derives them from authenticated context and intersects the requested operation with policy before searching. Apply document-level authorization before ranking so unauthorized candidates cannot influence scores, logs, caches, or model context.

Retrieved Content Is Untrusted Data

Documents, web pages, tickets, and tool outputs can contain prompt injection. Delimit evidence, preserve source identity, and instruct the model that retrieved text cannot change system policy or authorize tools. Keyword deletion is not a security boundary; enforce allowed tools, arguments, destinations, data volume, and side effects in code.

Provenance Must Survive Every Transformation

Query rewriting, chunking, reranking, summarization, and deduplication can sever provenance. Each evidence item needs an immutable source and revision, plus a mapping from generated claims to supporting excerpts. A URL alone is insufficient when content changes.

Build a Bounded Evidence-Control Loop

A production loop needs explicit state and transitions. “Continue until confident” is not a stopping policy because model confidence is neither calibrated nor enforceable.

State Contract

Record at least:

• request, user-visible goal, identity scope, and policy version;
• plan revision and evidence tasks;
• every tool name, normalized arguments, result IDs, duration, and failure;
• cumulative steps, retrievals, model calls, tokens, wall time, and cost;
• evidence coverage, conflicts, freshness, and authorization status;
• stop reason and the exact evidence used for each final claim.

Do not log full sensitive prompts or documents by default. Store redacted references and make raw-content access purpose-bound, time-limited, and audited.

Budget Contract

Set hard limits before execution:

NVIDIA's current Agentic RAG Blueprint defaults its agentic path off and recommends per-request enablement because extra planning and verification increase model calls and latency. Its exact implementation is vendor-specific, but the boundary is general: complex queries should pay for added control only when they need it.

Stop Contract

The loop should terminate with one of a small set of machine-readable reasons:

answered
clarification_required
insufficient_evidence
conflicting_evidence
authorization_denied
budget_exhausted
dependency_failed
policy_blocked

“Force generate with whatever context is available” is unsafe. If evidence is insufficient or contradictory, return the gap and safe next step. A partial answer is acceptable only when supported claims are clearly separated from unknowns.

A Runnable Trace Gate

The following dependency-free Python program validates an execution trace before a result can be promoted. It does not judge factual truth; it enforces deterministic invariants that an LLM must not override.

from __future__ import annotations

import argparse
import json
from pathlib import Path
from typing import Any


TERMINAL_REASONS = {
    "answered",
    "clarification_required",
    "insufficient_evidence",
    "conflicting_evidence",
    "authorization_denied",
    "budget_exhausted",
    "dependency_failed",
    "policy_blocked",
}


def is_number(value: Any) -> bool:
    return isinstance(value, (int, float)) and not isinstance(value, bool)


def validate(trace: dict[str, Any]) -> list[str]:
    errors: list[str] = []
    policy = trace.get("policy")
    steps = trace.get("steps")
    result = trace.get("result")
    if not isinstance(policy, dict):
        return ["policy must be an object"]
    if not isinstance(steps, list):
        return ["steps must be a list"]
    if not isinstance(result, dict):
        return ["result must be an object"]

    max_steps = policy.get("max_steps")
    max_tokens = policy.get("max_tokens")
    allowed_tools = policy.get("allowed_tools")
    allowed_sources = policy.get("allowed_source_scopes")
    if not isinstance(max_steps, int) or isinstance(max_steps, bool) or max_steps < 1:
        errors.append("policy.max_steps must be a positive integer")
    if not isinstance(max_tokens, int) or isinstance(max_tokens, bool) or max_tokens < 1:
        errors.append("policy.max_tokens must be a positive integer")
    if not isinstance(allowed_tools, list) or not all(
        isinstance(item, str) and item for item in allowed_tools
    ):
        errors.append("policy.allowed_tools must contain strings")
        allowed_tools = []
    if not isinstance(allowed_sources, list) or not all(
        isinstance(item, str) and item for item in allowed_sources
    ):
        errors.append("policy.allowed_source_scopes must contain strings")
        allowed_sources = []

    if isinstance(max_steps, int) and len(steps) > max_steps:
        errors.append("step budget exceeded")

    total_tokens = 0
    evidence_ids: set[str] = set()
    for index, step in enumerate(steps):
        if not isinstance(step, dict):
            errors.append(f"steps[{index}] must be an object")
            continue
        tool = step.get("tool")
        if tool not in allowed_tools:
            errors.append(f"steps[{index}] uses unauthorized tool {tool!r}")
        tokens = step.get("tokens", 0)
        if not is_number(tokens) or tokens < 0:
            errors.append(f"steps[{index}].tokens must be non-negative")
        else:
            total_tokens += tokens
        for item in step.get("evidence", []):
            if not isinstance(item, dict):
                errors.append(f"steps[{index}] has malformed evidence")
                continue
            evidence_id = item.get("id")
            scope = item.get("scope")
            if not isinstance(evidence_id, str) or not evidence_id:
                errors.append(f"steps[{index}] evidence needs an id")
            else:
                evidence_ids.add(evidence_id)
            if scope not in allowed_sources:
                errors.append(
                    f"steps[{index}] evidence uses unauthorized scope {scope!r}"
                )

    if isinstance(max_tokens, int) and total_tokens > max_tokens:
        errors.append("token budget exceeded")

    reason = result.get("stop_reason")
    if reason not in TERMINAL_REASONS:
        errors.append("result.stop_reason is invalid")
    citations = result.get("citation_ids", [])
    if not isinstance(citations, list) or not all(
        isinstance(item, str) and item for item in citations
    ):
        errors.append("result.citation_ids must contain strings")
        citations = []
    missing = set(citations) - evidence_ids
    if missing:
        errors.append(f"result cites unknown evidence: {sorted(missing)}")
    if reason == "answered" and not citations:
        errors.append("answered result must cite evidence")
    return errors


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("trace", type=Path)
    args = parser.parse_args()
    try:
        value = json.loads(args.trace.read_text(encoding="utf-8"))
    except (OSError, json.JSONDecodeError) as error:
        parser.error(str(error))
    if not isinstance(value, dict):
        parser.error("trace root must be an object")
    errors = validate(value)
    if errors:
        for error in errors:
            print(f"ERROR: {error}")
        raise SystemExit(1)
    print(f"trace passed: {args.trace}")


if __name__ == "__main__":
    main()

Run it with:

python agentic_rag_trace_gate.py trace.json

A trace can then be rejected for unauthorized tools, source-scope leakage, step or token budget overruns, unknown citations, or an invalid stop reason. Quality evaluation remains a separate stage.

Evaluate the System by Layer

Agentic RAG evaluation must isolate where a result improved or failed. A single LLM-as-judge score hides routing, retrieval, evidence, generation, and operational errors.

An LLM-as-judge can help score nuanced outputs, but calibrate it against human labels, test position and style bias, retain deterministic checks, and never let it authorize access or actions.

Build Query Slices

Use a versioned evaluation set with:

• single-source fact lookup;
• multi-source comparison;
• multi-hop questions with known evidence chains;
• ambiguous requests that should ask for clarification;
• unanswerable requests that should abstain;
• stale, conflicting, or revoked documents;
• tenant-bound questions and unauthorized distractors;
• prompt injection inside retrieved content;
• dependency timeout and partial-result cases.

Compare at least three variants: strong fixed RAG, routed RAG without iteration, and the full bounded loop. An ablation reveals whether query decomposition, correction, or verification earns its additional cost.

Measure Goodput, Not Raw Completion

Count a request as useful only if it meets the required answer-quality, citation, authorization, and latency objectives. More retrieval steps can raise an offline answer score while reducing user-visible goodput through timeouts or budget exhaustion.

The 2026 study on agent-orchestrated adaptive RAG reported that decomposition helped one structured domain but hurt ranking precision on a multi-hop benchmark, while reflection improved citation accuracy at substantial latency cost. Those results are not universal benchmarks; they are strong evidence that agentic components require per-slice ablation.

Observe Decisions Without Exposing Sensitive Content

A useful trace explains what the system did without defaulting to full prompt retention.

Capture:

• policy, prompt, model, retriever, index, and corpus revisions;
• selected tools and normalized arguments;
• source IDs, revisions, scores, filters, and authorization scopes;
• route, rewrite, correction, stop reason, and budget counters;
• citation mapping and output-schema validation;
• latency, tokens, retries, cache state, and errors by step.

Redact secrets and personal data before telemetry. Restrict access to raw evidence and set retention by purpose. Correlate traces with the agent observability workflow, but do not treat a generated rationale as an authoritative internal trace.

Roll Out as a Reversible Policy Change

Agentic RAG changes both answer behavior and resource consumption. Release it by query slice or tenant, not as an irreversible replacement for every request.

1. Freeze the baseline, corpus, index, and evaluation set.
2. Replay traces offline and inspect disagreements.
3. Shadow the agentic policy without returning its answer.
4. Canary only query slices with a measured failure mode the loop addresses.
5. Roll back automatically on quality, leakage, latency, token, or dependency-error regression.
6. Keep the fixed path available; do not require a rebuild to disable the loop.

Common Failure Modes

Treating a Grader as Ground Truth

A grader can misread a query, prefer verbose documents, or follow injected instructions. Use labeled retrieval data, deterministic metadata checks, and calibrated thresholds. Route uncertain cases to clarification or human review.

Retrying Without New Information

Paraphrasing the same query against the same index can consume budget without changing recall. Record the retrieval signature and require a meaningful change in source, filter, query, or evidence gap before retrying.

Losing Authorization During Decomposition

Subqueries inherit the original user's scope. A decomposed task must not broaden the tenant, time range, project, or data classification merely because a planner requested it.

Citing Retrieved but Unsupported Text

Presence in context is not support. Validate that each material claim is entailed by the cited excerpt and surface conflicts rather than selecting the most convenient source.

Mixing Retrieval and Side Effects

Searching an order and refunding it are different capabilities. Retrieval evidence may inform a proposal, but business actions require separate authorization, confirmation, idempotency, and outcome reconciliation.

Frequently Asked Questions

Is Agentic RAG always better than fixed RAG?

No. A fixed pipeline is easier to test, faster, and cheaper when one authorized search resolves the query. Use an agentic path only for query slices where dynamic source selection, decomposition, or correction produces a measured gain that survives latency and cost gates.

Does Agentic RAG eliminate hallucinations?

No. More retrieval and self-evaluation can still select irrelevant evidence, follow injected content, cite unsupported passages, or synthesize a wrong answer. Require claim-level support, unanswerable cases, calibrated abstention, and human review for high-impact decisions.

Should the agent search the public web when internal retrieval fails?

Only if policy explicitly permits it. Public web search changes data classification, source quality, freshness, licensing, privacy, and prompt-injection risk. Keep domains allowlisted, preserve citations, label external evidence, and never send sensitive internal context to a public search tool.

How many retrieval steps should Agentic RAG allow?

There is no universal number. Derive the limit from representative traces, the user-facing deadline, dependency capacity, and token budget. Set both a hard global cap and smaller per-tool or per-gap caps, then monitor budget-exhausted outcomes.

When should the system ask for clarification or abstain?

Clarify when a missing user choice can resolve ambiguity without exposing hidden data. Abstain when required evidence is absent, conflicting, unauthorized, stale beyond policy, or unavailable within the budget. Do not convert those states into confident prose.

Summary

Agentic RAG is useful when retrieval itself needs runtime decisions, but autonomy is not the objective. The objective is a better accepted answer under explicit quality, authorization, latency, and cost constraints. Treat retrieval as a typed and authorized tool, preserve evidence provenance, bound every loop, stop safely when evidence fails, evaluate each layer against a strong baseline, and roll out as a reversible policy.

References

• Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
• Corrective Retrieval Augmented Generation
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented LLMs through Question Complexity
• Microsoft: Develop an Agentic RAG Solution
• NVIDIA RAG Blueprint: Agentic RAG
• RAG Fundamentals and Evaluation
• Hybrid Search and Reranking