Transformer Architecture: Attention, Variants, Costs, and Evaluation

TL;DR

Transformers are neural-network architectures that combine attention, feed-forward layers, residual paths, normalization, and positional signals. This guide covers self-attention, masking, positional information, encoder-only, decoder-only, and encoder-decoder variants, plus the context, memory, latency, and evaluation trade-offs that determine whether a design fits a task.

Introduction

The 2017 paper "Attention Is All You Need" described an encoder-decoder Transformer. Its descendants are widely used in language, vision, audio, and multimodal systems, but the name alone does not establish quality or suitability. A production choice requires a dated model revision, representative task data, resource budget, and failure analysis.

In this guide, you'll learn:

• Core design principles of Transformer architecture
• Mathematical principles and intuitive understanding of self-attention
• How positional encoding enables models to understand sequence order
• How encoder-decoder architecture works
• Comparative analysis of Transformer vs RNN/LSTM
• The relationship between GPT, BERT and Transformer

What is Transformer

The original Transformer is an encoder-decoder sequence-to-sequence architecture. Its variants use attention over mask-permitted positions, feed-forward transformations, residual connections, normalization, and positional signals. Encoder-only models, decoder-only causal models, and encoder-decoder models have distinct information flow and task trade-offs.

What a Transformer Changes

Compared with recurrent designs, dense attention can expose more within-sequence computation to parallel hardware during training. It also creates direct, mask-controlled paths between positions. Those properties come with quadratic score and memory costs for dense attention, while autoregressive decoding remains sequential token by token. Evaluate architectures by target quality, context length, latency, peak memory, throughput, cost, safety, and operational complexity.

Self-Attention Mechanism Explained

Self-Attention is Transformer's core innovation. It allows the model to attend to all other positions in the sequence when processing each position.

Query, Key, Value Concepts

Self-attention uses three vectors to compute attention:

• Query: What information the current position wants to find
• Key: What information each position contains
• Value: The actual information each position transmits

import numpy as np

def scaled_dot_product_attention(Q, K, V):
    """
    Scaled dot-product attention computation
    Q: Query matrix (seq_len, d_k)
    K: Key matrix (seq_len, d_k)
    V: Value matrix (seq_len, d_v)
    """
    d_k = K.shape[-1]
    
    # Compute attention scores
    scores = np.matmul(Q, K.T) / np.sqrt(d_k)
    
    # Softmax normalization
    attention_weights = softmax(scores, axis=-1)
    
    # Weighted sum
    output = np.matmul(attention_weights, V)
    
    return output, attention_weights

def softmax(x, axis=-1):
    exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)

Attention Computation Formula

The mathematical expression for self-attention is:

Attention(Q, K, V) = softmax(QK^T / √d_k) V

Where d_k is the dimension of key vectors. Dividing by √d_k prevents dot products from becoming too large, which would cause softmax gradients to vanish.

Multi-Head Attention

To enable the model to attend to different types of information, Transformer uses Multi-Head Attention:

def multi_head_attention(Q, K, V, num_heads, d_model):
    """
    Multi-head attention mechanism
    """
    d_k = d_model // num_heads
    
    heads = []
    for i in range(num_heads):
        # Each head uses different linear projections
        Q_i = linear_projection(Q, d_k)
        K_i = linear_projection(K, d_k)
        V_i = linear_projection(V, d_k)
        
        head_i, _ = scaled_dot_product_attention(Q_i, K_i, V_i)
        heads.append(head_i)
    
    # Concatenate outputs from all heads
    concat = np.concatenate(heads, axis=-1)
    
    # Final linear projection
    output = linear_projection(concat, d_model)
    
    return output

Multi-head attention allows the model to simultaneously learn information from different representation subspaces—for example, one head focusing on grammatical structure while another focuses on semantic relationships.

Positional Encoding Principles

Since Transformer lacks recurrent structure, it cannot naturally perceive the position of elements in a sequence. Positional Encoding solves this problem.

Sinusoidal Positional Encoding

The original Transformer uses sine and cosine functions to generate positional encodings:

def positional_encoding(seq_len, d_model):
    """
    Generate sinusoidal positional encoding
    """
    position = np.arange(seq_len)[:, np.newaxis]
    div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
    
    pe = np.zeros((seq_len, d_model))
    pe[:, 0::2] = np.sin(position * div_term)  # Even dimensions
    pe[:, 1::2] = np.cos(position * div_term)  # Odd dimensions
    
    return pe

Advantages of this design:

• Each position has a unique encoding
• Model can learn relative positional relationships
• Can extrapolate to sequence lengths unseen during training

Learnable Positional Encoding

Modern models like BERT and GPT use learnable position embeddings:

class LearnablePositionalEncoding:
    def __init__(self, max_seq_len, d_model):
        # Position embeddings as trainable parameters
        self.position_embeddings = np.random.randn(max_seq_len, d_model) * 0.02

Encoder-Decoder Architecture

Transformer adopts an encoder-decoder architecture, a classic design for sequence-to-sequence tasks.

Encoder Structure

The encoder consists of N identical stacked layers, each containing:

1. Multi-Head Self-Attention Layer: Allows each position to attend to all positions in the input sequence
2. Feed-Forward Neural Network: Performs independent non-linear transformations at each position
3. Residual Connections and Layer Normalization: Stabilizes the training process

Decoder Structure

The decoder also consists of N stacked layers, but each layer has three sublayers:

1. Masked Multi-Head Self-Attention: Can only attend to already generated positions, preventing information leakage
2. Cross-Attention: Attends to encoder output to obtain source sequence information
3. Feed-Forward Neural Network: Same as encoder

Residual Connections and Layer Normalization

def transformer_sublayer(x, sublayer_fn):
    """
    Transformer sublayer: residual connection + layer normalization
    """
    # Sublayer computation
    sublayer_output = sublayer_fn(x)
    
    # Residual connection
    residual = x + sublayer_output
    
    # Layer normalization
    output = layer_norm(residual)
    
    return output

Transformer vs RNN/LSTM Comparison

Choosing Between Architectures

Transformers are often effective when attention-based context mixing and available tooling fit the workload. Recurrent, convolutional, state-space, retrieval, or hybrid designs can be preferable under streaming, memory, latency, or data constraints. Establish a baseline and evaluate the exact versions under the same task and resource budget.

Relationship Between GPT, BERT and Other Models

Modern large language models are all based on Transformer architecture but adopt different design choices:

GPT Series (Decoder-Only)

GPT uses Transformer's decoder component with autoregressive text generation:

• Training Objective: Predict next token
• Characteristics: Unidirectional attention, suitable for text generation
• Applications: Dialogue, writing, code generation

BERT (Encoder-Only)

BERT uses Transformer's encoder component with bidirectional attention:

• Training Objective: Masked Language Model (MLM) + Next Sentence Prediction
• Characteristics: Bidirectional context understanding
• Applications: Text classification, question answering, named entity recognition

T5 (Encoder-Decoder)

T5 retains the complete Transformer architecture:

• Training Objective: Text-to-text unified framework
• Characteristics: Flexible handling of various NLP tasks
• Applications: Translation, summarization, question answering

Practical Guide

Using Pre-trained Models

For many applications, a reviewed pre-trained model is a useful baseline. Verify its license, training-data policy, language coverage, context limit, and expected hardware before integrating it:

from transformers import AutoModel, AutoTokenizer

# Load pre-trained model
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

# Text encoding
text = "Transformer changed natural language processing"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

Fine-tuning Tips

1. Define a leakage-resistant train/validation/test or time-based split before tuning.
2. Start from a documented baseline and sweep learning rate, batch shape, schedule, and epochs for the target model and data.
3. Record model revision, tokenizer, preprocessing, random-seed policy, hardware, precision, metrics, and error slices.
4. Measure latency, memory, cost, safety failures, and rollback behavior before deployment.

Summary

Key points of Transformer architecture:

1. Attention and masks define which positions can interact and at what resource cost.
2. Positional signals encode order for attention-based sequence processing.
3. Variants differ: encoder-only, decoder-only, and encoder-decoder models support different information flows.
4. Operational constraints include context length, KV cache, batching, memory, latency, cost, safety, and version management.
5. Evidence comes from target-task evaluation, not from architecture labels or parameter count alone.

Transformer literacy is useful for reading modern model systems. Production decisions still require reproducible tests, failure analysis, and a rollback plan.

FAQ

What is the relationship between attention mechanism in Transformer and human attention?

Transformer's attention mechanism is a mathematical abstraction inspired by humans' ability to selectively focus on important information. In the model, attention weights represent the strength of correlation between different positions, similar to how humans focus on keywords when reading. However, this is a computational mechanism fundamentally different from biological neural system attention mechanisms.

Why does Transformer need positional encoding?

Because Transformer's self-attention mechanism is position-agnostic—it only considers relationships between elements without considering their positions in the sequence. Language understanding requires positional information ("dog bites man" and "man bites dog" have completely different meanings), so position information must be explicitly injected through positional encoding.

Which is better, GPT or BERT?

It depends on the specific task. GPT is suitable for generation tasks (writing, dialogue, code generation) because its autoregressive design naturally fits step-by-step generation. BERT is suitable for understanding tasks (classification, QA, information extraction) because its bidirectional attention better understands context. The modern trend shows GPT-class models can also perform understanding tasks well when scaled up.

Why is Transformer's computational complexity O(n²)?

Self-attention needs to compute attention scores between every pair of positions in the sequence. For a sequence of length n, n×n scores must be computed, hence O(n²) complexity. This is also the main bottleneck when processing very long texts, and much research focuses on developing linear-complexity attention variants.

How do I choose the right pre-trained model?

When choosing a pre-trained model, consider: 1) Task type (GPT-class for generation, BERT-class for understanding); 2) Language (choose language-specific pre-trained models for non-English tasks); 3) Model size (based on computational resources and latency requirements); 4) Domain (prefer domain-specific pre-trained models when available).