What is Tensor Parallelism?
Tensor Parallelism is an intra-layer model-parallel strategy that shards a tensor operation and its parameters across accelerators, then combines partial results with collective communication.
Quick Facts
| Created | Popularized for Transformers by Megatron-LM in 2019 |
|---|---|
| Specification | Official Specification |
How It Works
Pair Column and Row Shards
For Y = XW, Column-wise Parallelism splits W by output features, so each rank computes a different output slice from replicated input. Row-wise Parallelism splits W by input features, consumes matching input slices and combines partial sums. [Megatron-LM](https://arxiv.org/abs/1909.08053) pairs Column-wise and Row-wise projections so intermediate shards can stay local and communication occurs at selected boundaries. Attention head, KV head, hidden and vocabulary dimensions must satisfy the Runtime's divisibility and layout rules.
Put Collectives on the Topology
AllReduce, AllGather and ReduceScatter move activation data repeatedly through the model, not only once per request. Their cost depends on payload shape, TP degree, algorithm, link bandwidth, latency, contention and rank placement. Fast intra-node fabrics can make TP effective; crossing a slow network can erase compute savings. Current [vLLM scaling guidance](https://docs.vllm.ai/en/stable/serving/parallelism_scaling.html) recommends single-node TP when a model fits in one node and notes that Pipeline Parallelism may be preferable on nodes without a fast GPU interconnect.
Choose the Smallest Useful Degree
Start with no TP when the model and required KV capacity fit on one accelerator. Increase TP only enough to satisfy memory, supported head partitioning or latency goals, then use Data Parallel Replicas for additional request throughput when capacity permits. Compare candidate plans with identical model, precision, Context Length, Batch, arrival distribution and Cache state. Record per-rank weight and KV memory, collective bytes and time, imbalance, TTFT, TPOT, Throughput, Goodput, errors and output parity.
Key Characteristics
- Shards matrices and computations within each participating model layer
- Reduces per-rank weight storage while all ranks cooperate on the same request
- Uses collective operations such as AllReduce, AllGather or ReduceScatter
- Requires model dimensions and Attention head layouts compatible with the TP degree
- Places communication and straggler delay directly on the Forward Pass critical path
- Composes with Data, Pipeline, Sequence, Context and Expert Parallelism
Common Use Cases
- Serving or training a model whose layers do not fit on one accelerator
- Reducing single-request compute time when communication is sufficiently fast
- Keeping dense Transformer shards within one high-bandwidth GPU node
- Combining intra-node TP with inter-node Pipeline and replica-level Data Parallelism
- Evaluating model fit, collective cost and SLOs before choosing a TP degree
Example
Loading code...Frequently Asked Questions
How is Tensor Parallelism different from Data Parallelism?
Tensor Parallelism shards layers and makes several ranks cooperate on one request. Data Parallelism normally gives each Replica a complete model or model-parallel group and sends different requests or Batch shards to each Replica.
How is Tensor Parallelism different from Pipeline Parallelism?
Tensor Parallelism splits operations inside a layer, while Pipeline Parallelism assigns different layer ranges to stages. TP uses frequent collectives; PP transfers activations between stage boundaries and can introduce pipeline bubbles.
Does a larger Tensor Parallel degree always reduce latency?
No. More ranks reduce local matrix and weight work but increase collective participants, synchronization and topology sensitivity. Once communication dominates, adding GPUs can leave latency unchanged or make it worse.
Why is Tensor Parallelism usually kept within one node?
Collectives occur repeatedly inside the Forward Pass, so they benefit from low-latency, high-bandwidth GPU interconnects. Cross-node networking can work, but it needs explicit placement, RDMA and workload benchmarks rather than assuming linear scaling.
How should a Tensor Parallel degree be selected for inference?
Choose the smallest supported degree that fits weights, KV Cache and Runtime headroom or meets a measured latency goal. Verify dimension divisibility, per-rank memory, collective time, TTFT, TPOT, Throughput, Goodput and output parity on the target topology.