What is Transformer?
Transformer is a neural-network architecture first described in the 2017 paper "Attention Is All You Need." Its common variants combine attention, feed-forward layers, residual connections, normalization, and positional information to process token or other sequence representations.
Quick Facts
| Created | 2017 by Google (Vaswani et al.) |
|---|---|
| Specification | Official Specification |
How It Works
Transformer variants use attention to mix information across mask-permitted positions, feed-forward layers to transform representations, residual paths and normalization to stabilize optimization, and positional signals to encode order. Encoder-only models commonly support representation tasks, decoder-only models use causal masks for autoregressive generation, and encoder-decoder models use cross-attention for conditional generation. The original paper described an encoder-decoder variant; it is not a requirement for every Transformer. Dense attention can be parallelized during training but has sequence-length-dependent memory and compute costs. Modern systems combine architecture choices with kernel implementations, caching, quantization, batching, retrieval, and operational controls. Select a model using a dated checkpoint, task data, context length, latency, memory, cost, safety, and target-quality evaluation rather than assuming Transformers replace recurrent or convolutional models in every setting.
Key Characteristics
- Self-attention mechanism enabling parallel computation across all sequence positions
- Multi-head attention for capturing different types of relationships simultaneously
- Positional encoding to inject sequence order information into the model
- Supports encoder-only, decoder-only, and encoder-decoder variants for different tasks
- Layer normalization and residual connections for stable deep network training
- Can parallelize common training computations, while context length and decoding remain resource constraints
Common Use Cases
- Large language models (GPT, BERT, LLaMA) for text generation and understanding
- Machine translation and multilingual natural language processing
- Vision Transformers (ViT) for image classification and object detection
- Multimodal models combining text, image, and audio understanding
- Speech recognition and text-to-speech synthesis systems
Example
Loading code...Frequently Asked Questions
What is the self-attention mechanism in Transformers?
Self-attention computes weighted combinations of values using query-key compatibility scores over mask-permitted positions. A bidirectional encoder and a causal decoder use different masks, so a position does not always attend to every token. It can represent long-range interactions, but effective use still depends on context limits, training data, optimization, and evaluation.
What is the difference between encoder-only, decoder-only, and encoder-decoder Transformers?
Encoder-only models (like BERT) process input bidirectionally for understanding tasks. Decoder-only models (like GPT) generate text autoregressively, seeing only previous tokens. Encoder-decoder models (like T5) combine both for sequence-to-sequence tasks like translation, where the encoder processes input and decoder generates output.
Why are positional encodings necessary in Transformers?
Unlike RNNs that process sequences in order, Transformers process all positions simultaneously and have no inherent notion of sequence order. Positional encodings add information about token positions, either through learned embeddings or sinusoidal functions, enabling the model to understand word order and relative positions.
What is multi-head attention and why is it used?
Multi-head attention runs multiple attention operations in parallel, each with different learned projections. This allows the model to jointly attend to information from different representation subspaces at different positions. For example, one head might focus on syntax while another captures semantic relationships.
How do Vision Transformers (ViT) adapt the architecture for images?
Vision Transformers often convert image patches into token-like representations, add positional information, and process them with Transformer blocks. Specific designs differ in patching, hierarchy, pooling, pretraining, and objectives. Evaluate image classification, detection, or segmentation using the target data, resolution, latency, memory, and robustness requirements rather than treating one variant as universally state of the art.