How to Build an AI Agent: Production Architecture Guide

An AI agent is not simply an LLM with a long system prompt. It is a system in which a model can choose actions, observe results, update state, and continue until it reaches a stop condition. The model supplies judgment; the surrounding runtime supplies tools, permissions, memory, recovery, and evidence that the work was done correctly.

This distinction matters. Most failed agent projects do not fail because the model cannot reason. They fail because the system gives an uncertain model broad permissions, ambiguous tools, no durable state, and no reliable way to verify outcomes.

This guide develops a production-oriented architecture from first principles, then implements a small working agent in Python. It was last technically reviewed on July 16, 2026.

Key Takeaways

• Use a deterministic workflow when the path is known; use an agent only when the model must choose the path.
• Treat the agent loop as a state machine with explicit budgets and stop conditions.
• Design tools as narrow, typed capabilities. Tool output is untrusted input, even when it comes from your own infrastructure.
• Separate working state, conversation history, and long-term memory.
• Require human approval before consequential or irreversible actions.
• Evaluate trajectories and business outcomes, not just the final prose response.
• Start with one agent. Add specialists only when isolation, ownership, or parallel work creates measurable value.

What Is an AI Agent?

Anthropic makes a useful architectural distinction: a workflow follows code-defined paths, while an agent lets the model dynamically direct its process and tool use. Both are agentic systems, but their operating risk is different.

An agent is justified when all three conditions hold:

1. The task requires several steps or tools.
2. The correct sequence cannot be fully known in advance.
3. The environment provides feedback that lets the agent verify progress.

If a request can be solved by one model call, retrieval plus one model call, or a fixed directed acyclic graph, adding an agent usually increases latency, cost, and failure modes without increasing user value. Anthropic's guidance is explicit on this point: begin with the simplest solution and accept agentic complexity only when the performance benefit warrants the trade-off.

The Production Agent Architecture

A useful mental model is:

agent = model + harness + tools + environment

The model interprets the goal and chooses a next action. The harness owns instructions, state transitions, budgets, guardrails, and error handling. Tools expose bounded capabilities. The environment determines which files, networks, credentials, and services the process can access.

The loop is simple. Production behavior is not. Each arrow is a contract that needs a schema, timeout, ownership, and failure policy.

1. Goal and Policy

Translate a user request into an explicit execution contract:

• desired outcome;
• allowed and forbidden actions;
• data boundaries;
• cost and time budget;
• actions that require approval;
• definition of done.

"Resolve the customer's refund issue" is too vague. A safer contract is: "Inspect order and payment status, explain eligibility, and draft a refund. Never submit a refund above a policy-configured threshold or alter an order without approval." The threshold is an example policy value, not a universal business rule.

Prompt instructions are not an authorization system. Enforce permissions in application code and at the tool boundary.

2. Context and State

Context is the information presented to the model for its next decision. State is the authoritative record maintained by the runtime. Do not treat a growing chat transcript as both.

Retrieval should be deliberate. Load the smallest context that supports the next decision, attach source metadata, and prefer fresh reads for mutable business facts. A vector database is useful for semantic recall, but it must not become a shadow source of truth for balances, permissions, or order status.

3. Tools

A tool is a capability boundary, not a convenience wrapper around an internal API. Good tools make the correct action easy and dangerous actions difficult.

from typing import Literal
from pydantic import BaseModel, Field

class RefundDraft(BaseModel):
    order_id: str = Field(pattern=r"^ord_[a-zA-Z0-9]+$")
    amount_cents: int = Field(gt=0, le=10_000)
    reason: Literal["duplicate", "damaged", "not_received"]

Apply these rules:

• Give each tool one clear responsibility.
• Use typed inputs and structured outputs.
• Describe side effects and failure modes.
• Enforce identity, authorization, and tenant isolation inside the tool.
• Add timeouts, bounded retries, and idempotency keys.
• Return concise evidence, not entire database rows or unbounded logs.
• Treat tool descriptions and tool results as untrusted data.

The MCP specification makes the same security point for interoperable tools: users should understand exposed capabilities and retain control over sensitive invocations. MCP standardizes discovery and transport; it does not remove the need for application-level authorization.

4. The Agent Loop

The runtime repeats four operations:

1. Build context from the goal and current state.
2. Ask the model for a tool call, a final answer, or an escalation.
3. Execute the permitted action and capture the observation.
4. Update state and test a stop condition.

Every loop needs hard limits:

MAX_TURNS = 12
MAX_TOOL_CALLS = 20
MAX_WALL_SECONDS = 90
MAX_ESTIMATED_COST_USD = 0.50

These limits are illustrative starting points. Derive them from workload latency, provider pricing, retry behavior, and the business impact of interruption, then enforce them in the runtime rather than only describing them in a prompt.

Stop when the outcome is verified, the user must decide, the agent repeats without progress, or a budget is exhausted. "Keep trying until it works" is not a production policy.

5. Guardrails and Human Approval

Guardrails belong at several layers:

• Input: detect unsupported requests, prompt injection, and sensitive data.
• Model decision: restrict available tools based on user and task context.
• Tool input: validate schema, authorization, and business rules.
• Tool output: sanitize untrusted content and cap response size.
• Final output: verify required fields, citations, and policy compliance.

Use human approval for actions such as sending messages, moving money, deleting data, publishing changes, or granting access. The approval screen should show the exact proposed action and parameters, not a generic "allow agent" button.

6. Observability and Evaluation

Traditional service metrics remain necessary: latency, error rate, saturation, and cost. Agents also need semantic and trajectory-level evidence:

• model request and response metadata;
• selected tool and validated arguments;
• tool result, duration, and error category;
• state transition and retry reason;
• approval event and actor;
• final outcome and verification result.

Do not log secrets or unrestricted customer data. Redact before export and define retention separately for traces and product records.

Build an evaluation set from real, anonymized tasks: normal cases, ambiguous requests, tool failures, permission violations, prompt injection, and incomplete data. Run it on every prompt, model, tool-schema, or orchestration change. A final answer can sound excellent while the underlying action is wrong; outcome verification is therefore the release gate.

A Runnable Python Agent

The following example uses the official OpenAI Agents SDK because it exposes a small set of primitives: agents, function tools, guardrails, handoffs, sessions, and tracing. SDK APIs and model defaults change, so pin the dependency and check the current reference before deployment. The same design principles apply to other runtimes.

Install the SDK and configure the API key:

python -m venv .venv
source .venv/bin/activate
pip install openai-agents
export OPENAI_API_KEY="your-api-key"

Create support_agent.py:

import asyncio
from typing import Literal

from agents import Agent, Runner, function_tool
from pydantic import BaseModel


class OrderStatus(BaseModel):
    order_id: str
    status: Literal["processing", "shipped", "delivered"]
    refundable: bool


ORDERS = {
    "ord_1001": OrderStatus(
        order_id="ord_1001",
        status="delivered",
        refundable=True,
    )
}


@function_tool
def get_order_status(order_id: str) -> OrderStatus:
    """Return the current status and refund eligibility for one order."""
    if order_id not in ORDERS:
        raise ValueError("Order not found")
    return ORDERS[order_id]


agent = Agent(
    name="Order support",
    instructions=(
        "Help users understand order status and refund eligibility. "
        "Use get_order_status for order facts. Never claim that a refund "
        "was submitted; this agent can only explain or draft next steps. "
        "Ask for an order ID when it is missing."
    ),
    tools=[get_order_status],
)


async def main() -> None:
    result = await Runner.run(
        agent,
        "Can order ord_1001 still be refunded?",
        max_turns=6,
    )
    print(result.final_output)


if __name__ == "__main__":
    asyncio.run(main())

Run it:

python support_agent.py

Why this example is intentionally small:

• the tool reads from a bounded source and has no side effect;
• Pydantic provides a structured result;
• the instructions state what the agent cannot do;
• max_turns prevents an unbounded loop;
• the answer must be grounded in a fresh tool observation.

The in-memory order lookup is a read-only demonstration and intentionally omits authentication, tenant isolation, and concurrent updates. A production tool must receive the authenticated principal from trusted runtime context and enforce ownership against the source system before returning any order data.

To add refunds, do not simply expose issue_refund. Split the flow into draft_refund, an explicit approval checkpoint, and an idempotent submit_refund tool. Validate the authenticated user's ownership of the order inside both tools.

Choosing a Framework in 2026

Choose a runtime after defining the process, risk, and operational requirements. Framework popularity is not an architecture requirement.

LangChain v1 uses create_agent as its standard agent API; older tutorials based on create_react_agent should be treated as migration material. LangGraph focuses on orchestration capabilities such as persistence, durable execution, streaming, and human-in-the-loop rather than hiding prompts or architecture.

Single Agent or Multi-Agent?

Start with one agent and a small tool set. Add another agent only when at least one of these is true:

• a specialist needs a different permission boundary;
• context for one domain harms another domain's performance;
• independent work can run in parallel;
• separate teams own and evaluate separate capabilities.

Two common patterns are:

• Manager with agents as tools: one agent owns the user interaction and calls specialists for bounded tasks.
• Handoff: a triage agent transfers ownership and relevant context to a specialist.

Avoid multi-agent role-play that merely renames sequential prompts as "researcher," "analyst," and "writer." It increases model calls without necessarily improving evidence, control, or output quality.

Production Checklist

Before exposing an agent to users, verify:

• [ ] A deterministic workflow was considered first.
• [ ] Success, escalation, and stop conditions are explicit.
• [ ] Every tool has a narrow schema and documented side effects.
• [ ] Authorization is enforced in code, not only in prompts.
• [ ] External content and tool output are treated as untrusted.
• [ ] Consequential actions require specific human approval.
• [ ] Retries are bounded and write operations are idempotent.
• [ ] Working state can recover after process failure.
• [ ] Traces connect model decisions, tool calls, and business outcomes.
• [ ] Sensitive data is redacted and retention is defined.
• [ ] A representative evaluation set gates releases.
• [ ] Rollback or kill-switch behavior has been tested.

Frequently Asked Questions

What is the difference between an AI agent and RAG?

RAG retrieves context for a model. An agent controls a loop and can choose actions. An agent may use RAG as one tool, but retrieval alone does not make a system an agent.

Does an agent need long-term memory?

No. Many useful agents need only current-run state and fresh access to source systems. Add long-term memory when a concrete user benefit justifies its privacy, deletion, provenance, and staleness risks.

How do I prevent an agent from looping?

Set turn, tool-call, time, and cost limits. Detect repeated calls with equivalent arguments, require measurable progress after each observation, and escalate when the runtime cannot verify progress.

Should tool errors be returned to the model?

Return a bounded, actionable error category such as not_found, permission_denied, or temporary_failure. Do not expose stack traces, secrets, or raw infrastructure responses. Let the runtime decide which errors are retryable.

When should I use MCP?

Use MCP when tools or contextual resources need a standard interface across compatible hosts. MCP improves interoperability, but the host still owns user consent, server trust, authorization, and safe presentation of tool actions.

Conclusion

The essential unit of agent engineering is not the prompt; it is the controlled feedback loop. A production agent needs bounded tools, authoritative state, explicit stop conditions, human control over consequential actions, and evaluations tied to real outcomes.

Build the smallest loop that can solve the task. Make every action observable and recoverable. Only then increase autonomy.

Primary Sources

• Anthropic: Building Effective Agents
• OpenAI Agents SDK documentation
• LangGraph overview
• LangChain v1 migration guide
• Model Context Protocol specification