What is Context Compression?
Context Compression is a lossy transformation that reduces the context supplied to an LLM while preserving the evidence, constraints, and state required for a defined downstream task.
Quick Facts
| Full Name | Context Compression / Prompt Compression / Context Compaction |
|---|---|
| Specification | Official Specification |
How It Works
Context Compression reduces the representation of material that would otherwise enter a model request. Prompt Compression often refers to the same input-level operation, while Context Compaction commonly describes reducing accumulated conversation or Agent history. It is not ordinary file compression because the original wording may not be recoverable. It is also distinct from a larger Context Window, retrieval, Prompt Caching, and inference-time KV Cache compression: those change capacity, selection, repeated computation, or runtime state rather than necessarily producing a shorter request artifact.
Methods differ in what they preserve and where they run. Hard compression produces text that the target model can read directly: it may select documents or spans, remove duplicates, prune tokens, create an extractive digest, or generate an abstractive summary. Soft compression maps source content into learned continuous representations or memory slots and therefore depends on a compatible encoder and target model. Either family can be query-aware or task-agnostic. Retrieval and reranking can precede compression, but relevance ranking alone does not prove that every fact needed for the final task survived.
A production compressor needs an explicit loss budget, not only a target token count or compression ratio. Classify information before reduction: system and safety policy, authorization scope, user commitments, unresolved decisions, negation, numbers and units, identifiers, code or schema syntax, tool-call/result pairs, source provenance, temporal order, and conflicting evidence may require verbatim or structured preservation. Lower-priority narrative and duplicate material can be summarized or omitted. If protected content cannot fit, the safe result is an explicit overflow, retrieval, or human-review path rather than silent deletion.
Treat every compressed result as a derived artifact. Bind it to source IDs and immutable revisions, source spans where available, task or query identity, compressor and prompt revision, target model and Tokenizer, policy revision, tenant, method, input and output token counts, protected-field policy, and omitted-span record. Authorize sources before compression, keep tenant and data-class boundaries intact, and do not merge summaries across principals. Compression is not a sanitizer: it can preserve or amplify Prompt Injection, erase trust labels, or turn an untrusted instruction into authoritative-looking prose.
Compression has a lifecycle. Invalidate or regenerate an artifact when its source, permissions, task, policy, compressor, target model, or serialization changes. Preserve the authoritative source so users and evaluators can inspect the original evidence. Repeatedly summarizing a prior summary compounds omission and semantic drift; long-running systems should periodically rebuild from source records or a verified checkpoint instead of treating the latest summary as ground truth.
Evaluate compression against an uncompressed control on representative and adversarial workload slices. Token reduction is an efficiency metric, not a quality guarantee. Measure downstream task success, critical-evidence recall, policy-constraint recall, entity and number preservation, contradiction and unsupported-claim rates, citation entailment, abstention, end-to-end latency including compressor cost, and cost per successful task. Test exact quotes, tables, code, multilingual text, multi-hop evidence, conflicting sources, prompt injection, stale revisions, and cases where no compression is safe.
Release only a versioned policy with observable fallback behavior. Compare extractive, abstractive, token-pruning, and where applicable soft methods under the same source set, target model, token budget, and evaluation suite. Record whether each request used original or compressed context and why. Compression should improve a measured quality-cost frontier; when confidence or protected-information recall falls below its gate, use the original context, retrieve a narrower source set, split the task, ask for clarification, or reject the request.
Key Characteristics
- Lossy, task-bounded transformation: a shorter artifact is useful only if required evidence, constraints, and state survive
- Multiple representation families: hard text compression and model-specific soft or latent compression have different portability and auditability
- Explicit loss budget: protected facts, negation, numbers, permissions, provenance, and atomic tool interactions need preservation rules
- Versioned lineage: source revisions, compressor identity, task, target model, Tokenizer, tenant, and omitted spans define the artifact
- Measured quality-cost trade-off: token reduction must be evaluated with grounding, information preservation, task success, latency, and total cost
- Safe degradation: low-confidence or over-budget cases fall back to source retrieval, task splitting, original context, review, or rejection
Common Use Cases
- Compacting long conversation or Agent history while preserving decisions, constraints, tool outcomes, and open work
- Reducing retrieved RAG documents into query-specific, source-linked evidence spans
- Removing duplicate boilerplate while retaining conflicting evidence and provenance
- Building structured state from logs, tickets, or workflows without losing authorization and temporal semantics
- Reducing repeated input cost and prefill latency under workload-specific quality gates
Example
Loading code...Frequently Asked Questions
Are Context Compression, Prompt Compression, and Context Compaction the same?
They overlap but are not universal API synonyms. Prompt Compression usually shortens a model input, while Context Compaction often reduces accumulated conversation or Agent history. Context Compression is the broader engineering operation. A concrete system must still declare whether it selects text, summarizes, prunes tokens, creates structured state, or emits model-specific latent representations.
What is the difference between hard and soft context compression?
Hard compression emits shorter text through extraction, filtering, token pruning, or summarization, so it remains inspectable and portable to another text model. Soft compression encodes content into learned vectors or memory slots. It may achieve stronger compression but depends on a compatible encoder and target model and is harder for people to inspect, cite, or debug.
Why is a fluent summary not enough to prove safe compression?
Fluency does not show that negation, exceptions, numbers, permissions, conflicting evidence, code syntax, or earlier decisions survived. A summary can also add unsupported claims or elevate untrusted instructions. Keep source lineage and test critical-evidence, policy-constraint, entity, number, citation, contradiction, and task-level preservation against the original context.
How should Context Compression be evaluated?
Compare compressed and uncompressed controls using the same source set, query, target model, Tokenizer, and output policy. Measure token reduction, task success, grounding, critical-evidence recall, citation entailment, unsupported claims, abstention, end-to-end latency including the compressor, and cost per successful task across normal and adversarial slices.
When should a system avoid compression or fall back?
Avoid lossy compression when exact wording, legal or safety policy, code, schemas, numbers, or unresolved conflicts cannot be safely transformed. If protected content does not fit or preservation checks fail, retrieve a narrower source set, split the task, use the original context, request clarification or review, or reject the request instead of silently dropping information.