What is In-Context Learning?
In-Context Learning (ICL) is inference-time behavior in which a language model uses instructions, demonstrations, or other task evidence in its current context to shape predictions without updating model parameters.
Quick Facts
| Created | Identified in GPT-3 paper, 2020 |
|---|---|
| Specification | Official Specification |
How It Works
The prompt acts as an inference-time task contract
A useful ICL prompt separates instruction, demonstrations, query, and required output format. Demonstrations can communicate label semantics, edge cases, reasoning style, or serialization conventions. Keep the target answer absent from the context, use the same formatting path in evaluation and production, and distinguish task evidence from retrieved factual evidence.
Selection, order, and calibration are empirical choices
Performance can change with example relevance, class coverage, formatting, ordering, and label wording. Research has found that demonstrations may convey input distribution and format even when their labels contribute less than expected in some settings, while other tasks remain label-sensitive. Retrieve examples using task-relevant features, define deterministic tie-breaking, test multiple orders, and use calibration only when it improves a held-out metric.
The internal mechanism is not settled
Proposed explanations include pattern completion, implicit Bayesian inference, retrieval-like behavior, and transformer circuits that emulate learning algorithms in activations. Evidence for one mechanism under a synthetic task does not establish a universal account for all models and prompts. Operationally, ICL should be treated as conditional behavior with measured limits rather than proof that weights changed or a new algorithm was stored.
Production ICL trades flexibility for context and risk
Demonstrations consume tokens on every request, increase prefill work, and can expose sensitive examples or carry prompt injection. Version the prompt and example pool, filter untrusted content, separate instructions from data, and log selected example IDs rather than hidden personal text. Compare against zero-shot prompting, retrieval, rules, and fine-tuning on the same cases, including rare classes and distribution shifts.
Key Characteristics
- Adapts predictions through the current input without optimizer steps or weight updates
- Can use instructions, demonstrations, label definitions, constraints, and task evidence
- Ends when the relevant context is removed unless an external system stores it
- Is sensitive to model, template, example choice, ordering, formatting, and query distribution
- Consumes context capacity and prefill computation on every request
- Requires held-out evaluation because plausible demonstrations do not guarantee generalization
Common Use Cases
- Defining a new classification label set for a request without training a new checkpoint
- Demonstrating a required JSON, table, or domain-specific output convention
- Selecting similar reviewed cases to guide support, moderation, or extraction decisions
- Testing whether a task is learnable from examples before investing in fine-tuning
- Switching bounded tasks at runtime while preserving one shared base model
Example
Loading code...Frequently Asked Questions
How is in-context learning different from fine-tuning?
ICL changes the request context and leaves model weights unchanged, so examples must usually be supplied again and consume input tokens. Fine-tuning uses an optimizer to update parameters and produces a new checkpoint or adapter. Compare both on the same held-out cases, including operational cost, privacy, update frequency, and rollback needs.
How many demonstrations should an ICL prompt contain?
There is no universal count. The useful number depends on model, task, class coverage, example length, context budget, and selection method. Start with zero-shot as a baseline, add reviewed examples incrementally, and plot held-out quality against tokens, latency, and cost. Stop when additional demonstrations no longer justify their overhead.
Do demonstration labels and order matter?
They can, but sensitivity varies by model and task. Examples communicate format and input distribution as well as labels, and ordering can interact with recency, class balance, or prompt structure. Use valid labels, deterministic selection and ordering, then evaluate alternative orders instead of relying on a fixed rule such as always placing the closest example last.
Does ICL mean the model learns a new skill permanently?
No. ICL normally changes predictions only while the instruction and demonstrations remain in the active context. Persistence requires an external memory, prompt store, retrieval system, adapter, or weight update. The exact internal mechanism is still researched, so activation-level adaptation should not be described as confirmed parameter learning.
What can go wrong when demonstrations come from users or retrieval?
They can contain private data, incorrect labels, stale policy, prompt injection, adversarial formatting, or examples too similar to evaluation answers. Apply access control and provenance, redact sensitive fields, isolate instructions from data, validate labels, deduplicate against test sets, cap context usage, and log selected example IDs for audits.