Context Engineering: Selection, Evidence, and State for LLM Systems

A Useful Definition

Context engineering is the deliberate construction of the input state for a model or Agent. It includes:

task + instructions + evidence + state + tool results + metadata
       -> selection/order -> model call -> observable outcome

It complements prompt engineering rather than replacing it. A short instruction can still be critical, and a large evidence set can still be irrelevant. The goal is not to maximize context length; it is to provide the smallest trustworthy context that supports the task.

Start With a Context Contract

Define the contract before choosing a vector database or memory store:

This contract is not a prompt-only rule. A retrieval layer and tool executor must enforce tenant scope, access policy, and output bounds.

Context Layers

Separate information by trust and lifetime:

Treat user text, documents, comments, memory, and tool results as untrusted content. They may contain instructions aimed at changing the task, exfiltrating data, or bypassing policy.

Selection and Ordering

Selection should be task-specific. Useful signals include:

• lexical and semantic relevance;
• authority and source trust;
• recency and validity interval;
• tenant and object permissions;
• diversity and redundancy;
• evidence coverage of required claims;
• size, latency, and cost.

Ordering also matters. Put task constraints and the evidence needed for the next decision where the model can reliably use them. Do not assume a fixed “top K” is portable across domains. Log the selection reason and item revisions for evaluation.

Retrieval Is Not Authorization

Retrieval must filter by access before ranking. A high similarity score cannot grant access to a document. A citation cannot prove that the caller owns the object.

For every item, preserve enough metadata to answer:

source -> revision -> owner/tenant -> permission decision
      -> retrieval reason -> context position -> model outcome

If permissions change, propagate the change to indexes, caches, summaries, memories, backups, and evaluation fixtures.

Compression With Evidence

Compression can remove repeated history, logs, markup, or irrelevant fields. It can also remove negation, qualifiers, exceptions, or the source of a claim.

Use a compression record:

{
  "source_ids": ["doc-17@rev-4", "doc-22@rev-9"],
  "summary_revision": "sum-3",
  "claims": [
    {
      "id": "c1",
      "text": "The rollout is paused for tenant B.",
      "source_spans": ["doc-17#L88-L94"]
    }
  ],
  "omitted": ["raw debug logs"],
  "expires_at": "recorded-by-policy"
}

Keep source links and uncertainty. Evaluate whether compression changes task success, unsupported claims, refusal behavior, and latency. There is no universal compression ratio or quality guarantee.

Persistence and Memory

Persist only information with a defined purpose:

• validity and observed time;
• source and confidence;
• subject and tenant scope;
• sensitivity and retention;
• supersession and deletion;
• who or what may read it.

A checked-in instruction file, a conversation summary, and semantic memory have different ownership and failure modes. Never let memory silently override current authorization or a newer source.

Token, Latency, and Cost Budgets

A context budget should include more than model window size:

input tokens + output tokens
+ retrieval latency + reranking latency
+ storage/index cost + cache behavior
+ retries and downstream tool time

Measure p50/p95/p99 latency and cost per successful task. Prompt caching may help when a provider and workload support it, but cache keys, invalidation, privacy, and billing semantics must be verified. Compression may reduce input size while adding a model call and losing evidence.

Context Security

Use defense in depth:

• classify and minimize data before retrieval;
• enforce identity and object permissions outside the model;
• mark provenance and taint untrusted content;
• isolate tools and restrict egress;
• bound retrieved text, tool results, and memory writes;
• require approval for destructive or external actions;
• redact telemetry and define deletion propagation;
• test direct and indirect prompt injection, poisoning, stale state, and cross-tenant access.

Context is an input channel, not a policy boundary. Do not place secrets, bearer tokens, or unrestricted production dumps in a context file.

Evaluate Context Changes

Changing retrieval, ordering, compression, memory, or a system instruction can change behavior. Keep a versioned evaluation set:

Use deterministic oracles where possible and calibrated human/model review for open-ended claims. Do not treat a model score as proof that context is safe or complete.

A Small Context Builder

The builder should make selection and limits explicit:

from dataclasses import dataclass
from typing import Iterable

@dataclass(frozen=True)
class Evidence:
    source_id: str
    revision: str
    text: str
    tenant_id: str
    allowed: bool
    relevance: float

def build_context(
    task: str,
    evidence: Iterable[Evidence],
    *,
    tenant_id: str,
    max_chars: int,
) -> list[Evidence]:
    candidates = [
        item for item in evidence
        if item.allowed and item.tenant_id == tenant_id
    ]
    ordered = sorted(candidates, key=lambda item: item.relevance, reverse=True)
    selected: list[Evidence] = []
    used = 0
    for item in ordered:
        if used + len(item.text) > max_chars:
            break
        selected.append(item)
        used += len(item.text)
    return selected

This is a structural fragment, not a complete retriever. Production code still needs authentication, object-level authorization, robust token accounting, stale-data handling, provenance, pagination, and tests.

Common Failure Modes

• treating the context window as a guarantee of comprehension;
• saying prompt engineering is dead;
• retrieving before applying access control;
• trusting similarity, citations, or memory as authorization;
• compressing away exceptions and source spans;
• persisting user data without purpose, retention, and deletion;
• allowing documents or tool results to inject instructions;
• using a fixed token limit, top-K, cache policy, or savings percentage everywhere;
• measuring answer fluency without evidence support;
• placing model-controlled identity, ownership, pricing, or roles in context.

Practical Checklist

• [ ] Define task, audience, evidence, freshness, provenance, sensitivity, budget, and deletion.
• [ ] Separate instructions, working state, retrieved evidence, durable memory, and environment.
• [ ] Apply permissions before ranking or injecting content.
• [ ] Record source, revision, selection reason, and validity.
• [ ] Preserve source spans and uncertainty through compression.
• [ ] Bound tokens, latency, tool results, memory writes, and telemetry.
• [ ] Keep secrets and unrestricted production data out of default context.
• [ ] Test injection, poisoning, stale state, deletion, and cross-tenant cases.
• [ ] Evaluate context changes with task, safety, and operational evidence.
• [ ] Keep model scores subordinate to deterministic policy and business oracles.

Conclusion

Context engineering is the disciplined management of what an LLM or Agent can use for a task: selecting relevant evidence, preserving provenance, controlling state, respecting permissions, and measuring outcomes. It is not a contest to fill the largest window or a replacement for prompt design. Build a context contract first, enforce security outside the model, and let workload evidence decide whether retrieval, compression, caching, or persistence actually helps.

Primary Sources

• NIST AI Risk Management Framework
• OWASP Top 10 for LLM Applications
• W3C Trace Context
• Model Context Protocol Specification