What is Transformer?
Transformer is a neural-network architecture that repeatedly updates a sequence of vectors with attention and position-wise feed-forward sublayers connected by residual paths and normalization. A complete model also defines positional information, visibility masks, input and output representations, a training objective, and an inference procedure.
Quick Facts
| Created | 2017 by Google (Vaswani et al.) |
|---|---|
| Specification | Official Specification |
How It Works
Block structure
A Transformer block preserves a residual-stream shape such as [batch, tokens, model_dimension]. Attention mixes information across positions allowed by a mask. A feed-forward network then transforms each position independently with shared parameters. Residual addition and normalization connect both sublayers. Pre-norm and post-norm are different forward functions and must match the checkpoint.
Position and visibility
Attention alone does not encode sequence order. Models add absolute embeddings, relative biases, rotary position embeddings, or another position mechanism. Visibility comes from causal, padding, cross-attention, or application masks. Position IDs, mask polarity, cached offsets, and maximum trained context are runtime contracts; accepting a longer tensor does not prove reliable length generalization.
Architecture families
Encoder-only models usually create bidirectional representations from observed input. Decoder-only models apply a causal mask and commonly predict the next token. Encoder-decoder models first encode a source sequence, then generate a target with causal self-attention and cross-attention. These are information-flow patterns, not guarantees that one family is best for every task.
Training and inference
Teacher-forced causal training can process known target positions in parallel behind a mask. Autoregressive generation remains sequential because each next token depends on the token selected immediately before it. A KV cache reuses earlier Keys and Values, reducing recomputation while adding memory, position, eviction, reuse, and isolation requirements.
Cost and verification
Dense attention score work grows quadratically with sequence length, while projections and feed-forward work also depend heavily on model width. KV-cache bytes scale with batch, cached tokens, layers, KV heads, head dimension, and precision. Reliable implementations test shapes, masks, finite outputs, position indices, train/eval behavior, and parity between full-sequence and cached decoding.
Key Characteristics
- Combines attention and position-wise feed-forward sublayers inside a residual stack
- Uses explicit positional information because attention alone does not represent order
- Defines information visibility through causal, padding, cross-attention, or application masks
- Supports encoder-only, decoder-only, and encoder-decoder information flows
- Separates parallelizable training or prefill work from sequential autoregressive token selection
- Has measurable trade-offs across quality, context length, parameter count, KV-cache memory, latency, and throughput
Common Use Cases
- Bidirectional representation models for classification, retrieval, and token-level prediction
- Autoregressive language or code generation with causal masking and KV-cache state
- Conditional generation such as translation, summarization, and structured transformation
- Vision, audio, and multimodal systems that map observations into token-like representations
- Sequence-processing systems evaluated under explicit quality, latency, memory, and reliability budgets
Example
Loading code...Frequently Asked Questions
What is the difference between attention and a Transformer?
Attention is an operation that mixes Value vectors according to Query-Key compatibility over permitted positions. A Transformer is the complete residual architecture around that operation, including embeddings, positional signals, feed-forward layers, normalization, masks, output heads, and a training objective.
Do all Transformers have an encoder and a decoder?
No. The original Transformer used both. Encoder-only models retain an encoder-style bidirectional stack, decoder-only models use a causal stack, and encoder-decoder models keep separate source and target stacks connected through cross-attention. The actual mask and objective matter more than the label.
Why does a Transformer need positional information?
Attention over content vectors does not by itself distinguish sequence order. Positional information makes order available through absolute embeddings, relative biases, rotary transformations, or another scheme. The method and position-index rules must match the trained checkpoint.
Why is Transformer attention called quadratic?
Dense self-attention forms a score for each permitted pair of sequence positions, so the score and Value-mixing work grows with the square of sequence length. Projection and feed-forward work have different scaling, and optimized kernels can reduce memory traffic without removing all pairwise arithmetic.
What does a KV cache change in a Transformer?
During autoregressive inference, a KV cache stores Keys and Values for earlier tokens so they are not projected again at every step. It reduces repeated computation but consumes memory proportional to cached tokens, layers, KV heads, head dimension, batch size, and scalar precision.