What is Attention Mechanism?
Attention Mechanism is a differentiable operation that scores permitted Query-Key pairs, normalizes those scores, and uses the resulting weights to combine Value vectors into context-dependent outputs.
Quick Facts
| Created | 2014 by Bahdanau et al., popularized in 2017 by Vaswani et al. |
|---|---|
| Specification | Official Specification |
How It Works
Self-Attention, Cross-Attention, and Visibility
Self-attention derives Q, K, and V from the same representation stream. Cross-attention takes Queries from one stream and Keys and Values from another. A causal mask prevents a generated position from reading future positions, while a padding mask excludes artificial batch padding. Mask polarity and broadcasting are API contracts that must be tested explicitly.
Multi-Head Design and Inference
Multi-head attention applies several learned projection sets, computes attention in parallel, concatenates head outputs, and applies an output projection. MQA and GQA reduce the number of distinct KV heads to lower incremental decoding bandwidth and KV cache capacity, but they are trained architecture choices rather than lossless runtime switches.
Complexity and Interpretation
Dense attention evaluates target-source position pairs, so score work grows with target length times source length. Exact fused kernels such as FlashAttention reduce intermediate storage and memory traffic without making dense pairwise arithmetic universally linear. Attention weights describe one layer and head's routing coefficients; they are useful diagnostics but do not alone establish the causal reason for a final prediction.
Key Characteristics
- Separates Query-Key compatibility scoring from Value aggregation
- Produces one normalized distribution over permitted Keys for each Query row
- Supports self-attention and cross-attention through different Q/K/V sources
- Uses causal, padding, local, or architecture-specific masks to define visibility
- Extends to multi-head, multi-query, and grouped-query layouts
- Requires shape, mask, numerical, quality, and performance tests
Common Use Cases
- Contextual token interaction in Transformer encoders and decoder-only language models
- Causal next-token prediction without exposing future target positions
- Cross-modal fusion between text, image, audio, or structured representations
- Encoder-decoder alignment for translation, transcription, and conditional generation
- Long-sequence systems that combine exact kernels, sparse patterns, or KV caching under measured trade-offs
Example
Loading code...Frequently Asked Questions
What does an attention mechanism do?
For each Query, attention scores the permitted Keys, normalizes the scores, and computes a weighted combination of the corresponding Values. This creates a context-dependent representation. The weights are learned behavior inside a larger model, not a symbolic database lookup or a guaranteed explanation.
What is the difference between self-attention and cross-attention?
Self-attention derives Query, Key, and Value representations from the same stream, such as one token sequence. Cross-attention uses Queries from one stream and Keys and Values from another, such as decoder states reading encoder outputs or text positions reading image features.
Why does attention use Query, Key, and Value projections?
Separate learned projections let the model use one representation space for matching Queries to Keys and another for the content carried by Values. The projections are optimized through the task loss. Their retrieval analogy is useful, but no architecture rule assigns fixed human-readable semantics to each dimension.
Why are causal and padding masks different?
A causal mask enforces time or generation order by blocking future positions for each Query. A padding mask removes artificial tokens added to make batch lengths equal. They may be combined, but frameworks differ in boolean polarity, shape, and broadcasting, so implementations need explicit mask tests.
Is attention always quadratic?
Dense self-attention evaluates every permitted target-source position pair, producing quadratic score terms when target and source lengths are equal. FlashAttention reduces memory traffic and materialized intermediates without changing the exact pairwise operation. Sparse, local, or approximate attention can reduce pair counts by changing visibility or approximation.
Can attention weights explain a model output?
They can show routing patterns for a specific head, layer, input, and run, but they omit Value vectors, output projections, residual paths, other heads, and later layers. Treat heatmaps as diagnostic evidence and use interventions such as ablation or controlled counterfactuals to test causal claims.