Context Engineering: Selection, Evidence, and State for LLM Systems

A Useful Definition

Context engineering is the deliberate construction of the input state for a model or Agent. It includes:

task + instructions + evidence + state + tool results + metadata
       -> selection/order -> model call -> observable outcome

It complements prompt engineering rather than replacing it. A short instruction can still be critical, and a large evidence set can still be irrelevant. The goal is not to maximize context length; it is to provide the smallest trustworthy context that supports the task.

Start With a Context Contract

Define the contract before choosing a vector database or memory store:

This contract is not a prompt-only rule. A retrieval layer and tool executor must enforce tenant scope, access policy, and output bounds.

Context Layers

Separate information by trust and lifetime:

Treat user text, documents, comments, memory, and tool results as untrusted content. They may contain instructions aimed at changing the task, exfiltrating data, or bypassing policy.

Four Operations: Write, Select, Compress, Isolate

Context engineering is easier to reason about as four different operations:

Choose the operation from the failure you need to control. Do not create a sub-Agent merely to shorten a prompt, write every transient observation into durable memory, or compress evidence that can be freshly retrieved from its source.

Selection and Ordering

Selection should be task-specific. Useful signals include:

• lexical and semantic relevance;
• authority and source trust;
• recency and validity interval;
• tenant and object permissions;
• diversity and redundancy;
• evidence coverage of required claims;
• size, latency, and cost.

Ordering also matters. Put task constraints and the evidence needed for the next decision where the model can reliably use them. Do not assume a fixed “top K” is portable across domains. Log the selection reason and item revisions for evaluation.

Chroma's Context Rot study evaluated 18 models under controlled tasks and found non-uniform degradation as input length and distractors increased, including cases where relevant information remained present. That result does not establish a universal safe token threshold. It means each application should measure a task-level quality curve across realistic context lengths, evidence positions, distractor mixes, and model revisions instead of assuming that “within the window” means “equally usable.”

Retrieval Is Not Authorization

Retrieval must filter by access before ranking. A high similarity score cannot grant access to a document. A citation cannot prove that the caller owns the object.

For every item, preserve enough metadata to answer:

source -> revision -> owner/tenant -> permission decision
      -> retrieval reason -> context position -> model outcome

If permissions change, propagate the change to indexes, caches, summaries, memories, backups, and evaluation fixtures.

Compression With Evidence

Compression can remove repeated history, logs, markup, or irrelevant fields. It can also remove negation, qualifiers, exceptions, or the source of a claim.

Use a compression record:

{
  "source_ids": ["doc-17@rev-4", "doc-22@rev-9"],
  "summary_revision": "sum-3",
  "claims": [
    {
      "id": "c1",
      "text": "The rollout is paused for tenant B.",
      "source_spans": ["doc-17#L88-L94"]
    }
  ],
  "omitted": ["raw debug logs"],
  "expires_at": "recorded-by-policy"
}

Keep source links and uncertainty. Evaluate whether compression changes task success, unsupported claims, refusal behavior, and latency. There is no universal compression ratio or quality guarantee.

Compaction, tool-result clearing, and memory solve different problems:

Clearing a tool result is not the same as summarizing the whole run, and memory is not overflow storage. Prefer re-reading mutable facts from the system of record over carrying an old summary forward.

Persistence and Memory

Persist only information with a defined purpose:

• validity and observed time;
• source and confidence;
• subject and tenant scope;
• sensitivity and retention;
• supersession and deletion;
• who or what may read it.

A checked-in instruction file, a conversation summary, and semantic memory have different ownership and failure modes. Never let memory silently override current authorization or a newer source.

Token, Latency, and Cost Budgets

A context budget should include more than model window size:

input tokens + output tokens
+ retrieval latency + reranking latency
+ storage/index cost + cache behavior
+ retries and downstream tool time

Measure p50/p95/p99 latency and cost per successful task. Prompt caching may help when a provider and workload support it, but cache keys, invalidation, privacy, and billing semantics must be verified. Compression may reduce input size while adding a model call and losing evidence.

Context Security

Use defense in depth:

• classify and minimize data before retrieval;
• enforce identity and object permissions outside the model;
• mark provenance and taint untrusted content;
• isolate tools and restrict egress;
• bound retrieved text, tool results, and memory writes;
• require approval for destructive or external actions;
• redact telemetry and define deletion propagation;
• test direct and indirect prompt injection, poisoning, stale state, and cross-tenant access.

Context is an input channel, not a policy boundary. Do not place secrets, bearer tokens, or unrestricted production dumps in a context file.

Evaluate Context Changes

Changing retrieval, ordering, compression, memory, or a system instruction can change behavior. Keep a versioned evaluation set:

Use deterministic oracles where possible and calibrated human/model review for open-ended claims. Do not treat a model score as proof that context is safe or complete.

Test Invariants Before and After Compaction

A shorter context is acceptable only if it preserves behaviorally important invariants. Build probes that compare the original and compacted state:

• forbidden actions, tenant boundaries, and approval requirements remain unchanged;
• negations, exceptions, deadlines, units, and unresolved questions remain explicit;
• each decision-critical claim still points to a source revision and freshness state;
• completed side effects, idempotency keys, and pending operations cannot be executed twice;
• missing or conflicting evidence still triggers abstention or escalation;
• deletion and supersession markers survive into every derived store.

Measure task outcome and policy behavior, not summary similarity alone. A compact summary can be lexically close while dropping the single exception that controls a financial or security decision.

A Small Context Builder

The builder should make selection and limits explicit:

package main

import (
	"fmt"
	"sort"
	"unicode/utf8"
)

type Evidence struct {
	SourceID  string
	Revision  string
	Text      string
	TenantID  string
	Allowed   bool
	Relevance float64
}

func buildContext(evidence []Evidence, tenantID string, maxChars int) []Evidence {
	candidates := make([]Evidence, 0, len(evidence))
	for _, item := range evidence {
		if item.Allowed && item.TenantID == tenantID {
			candidates = append(candidates, item)
		}
	}
	sort.SliceStable(candidates, func(left, right int) bool {
		if candidates[left].Relevance == candidates[right].Relevance {
			return candidates[left].SourceID < candidates[right].SourceID
		}
		return candidates[left].Relevance > candidates[right].Relevance
	})

	selected := make([]Evidence, 0, len(candidates))
	used := 0
	for _, item := range candidates {
		size := utf8.RuneCountInString(item.Text)
		if size > maxChars-used {
			continue
		}
		selected = append(selected, item)
		used += size
	}
	return selected
}

func main() {
	evidence := []Evidence{
		{SourceID: "doc-17", Revision: "4", Text: "rollout paused", TenantID: "a", Allowed: true, Relevance: 0.95},
		{SourceID: "doc-22", Revision: "9", Text: "private note", TenantID: "b", Allowed: true, Relevance: 0.99},
		{SourceID: "doc-31", Revision: "2", Text: "owner approved", TenantID: "a", Allowed: true, Relevance: 0.80},
	}
	for _, item := range buildContext(evidence, "a", 30) {
		fmt.Printf("%s@%s\n", item.SourceID, item.Revision)
	}
}

This is a structural fragment, not a complete retriever. Production code still needs authentication, object-level authorization, robust token accounting, stale-data handling, provenance, pagination, and tests.

Common Failure Modes

• treating the context window as a guarantee of comprehension;
• saying prompt engineering is dead;
• retrieving before applying access control;
• trusting similarity, citations, or memory as authorization;
• compressing away exceptions and source spans;
• persisting user data without purpose, retention, and deletion;
• allowing documents or tool results to inject instructions;
• using a fixed token limit, top-K, cache policy, or savings percentage everywhere;
• measuring answer fluency without evidence support;
• placing model-controlled identity, ownership, pricing, or roles in context.

Practical Checklist

• [ ] Define task, audience, evidence, freshness, provenance, sensitivity, budget, and deletion.
• [ ] Separate instructions, working state, retrieved evidence, durable memory, and environment.
• [ ] Apply permissions before ranking or injecting content.
• [ ] Record source, revision, selection reason, and validity.
• [ ] Preserve source spans and uncertainty through compression.
• [ ] Bound tokens, latency, tool results, memory writes, and telemetry.
• [ ] Keep secrets and unrestricted production data out of default context.
• [ ] Test injection, poisoning, stale state, deletion, and cross-tenant cases.
• [ ] Evaluate context changes with task, safety, and operational evidence.
• [ ] Keep model scores subordinate to deterministic policy and business oracles.

Frequently Asked Questions

What is context engineering?

It is the design and operation of the instructions, evidence, state, tool results, and metadata supplied to a model for a task. It complements prompt engineering and includes selection, provenance, permissions, budgets, persistence, and evaluation.

Is RAG part of context engineering?

Yes, retrieval can be one component. Context engineering also decides what may be retrieved, how evidence is ordered and cited, when it expires, and whether it should be compressed, isolated, persisted, or omitted.

Does more context improve answers?

Not reliably. Long context can add stale facts, contradictions, distractors, latency, and attack surface. Measure task quality across realistic lengths and evidence positions for the exact model revision.

When should context be compressed?

Compress when an active trajectory exceeds its budget and important invariants can be tested. Preserve constraints, exceptions, provenance, pending actions, and side-effect state; clear transient tool output or retrieve fresh facts when those operations are safer.

Is a project instruction file a security control?

No. It is input to a probabilistic model. Trusted code and infrastructure must enforce identity, object authorization, secret handling, network egress, approvals, and side effects.

Conclusion

Context engineering is the disciplined management of what an LLM or Agent can use for a task: selecting relevant evidence, preserving provenance, controlling state, respecting permissions, and measuring outcomes. It is not a contest to fill the largest window or a replacement for prompt design. Build a context contract first, enforce security outside the model, and let workload evidence decide whether retrieval, compression, caching, or persistence actually helps.

Primary Sources

• NIST AI Risk Management Framework
• OWASP Top 10 for LLM Applications
• Anthropic: Effective Context Engineering for AI Agents
• Anthropic Cookbook: Context Management
• Chroma Research: Context Rot
• Model Context Protocol Specification