What is Mixture of Experts?
Mixture of Experts (MoE) is a conditional-computation architecture in which a learned or fixed routing policy selects a subset of expert subnetworks for each input and combines their outputs.
Quick Facts
| Full Name | Mixture of Experts (MoE) |
|---|---|
| Created | 1991 by Jacobs et al., popularized in LLMs since 2022 |
| Specification | Official Specification |
How It Works
Follow Route, Dispatch, Compute, and Combine
A Router scores each Token state, selects Top-k experts, dispatches states to those expert FFNs, and combines returned outputs with routing weights. The [Switch Transformer paper](https://arxiv.org/html/2101.03961) demonstrates Top-1 routing and explicit Expert Capacity, but Top-k, shared experts, dropped-token behavior and auxiliary losses vary by architecture. An Expert is usually a subnetwork inside a layer, not an independently callable domain model.
Make Capacity and Load Policy Observable
Record assignments per Expert, maximum-to-mean load, Capacity, dropped or rerouted Tokens, Padding, Router entropy and auxiliary-loss contribution by layer and Batch. Current [Megatron Core MoE documentation](https://docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/moe.html) supports dropless and capacity-based dropping modes plus several balancing strategies. These are selectable implementation policies, not one universal MoE algorithm.
Benchmark the Distributed System End to End
Total parameters describe stored capacity; active parameters describe only part of the selected compute path. Neither determines latency, memory or quality because shared layers, KV Cache, routing, Token permutation, padding, Kernel efficiency and communication remain. [TensorRT-LLM Expert Parallelism](https://nvidia.github.io/TensorRT-LLM/1.1.0rc2/advanced/expert-parallelism.html) distinguishes full-Expert placement from Tensor Parallel sharding. Compare Dense and MoE candidates on the same Checkpoint task, quality gates, Runtime, topology and workload.
Key Characteristics
- Uses conditional computation to activate a subset of subnetworks for each input
- Commonly replaces selected Transformer FFN blocks while other components remain dense or shared
- Routes batched token states, groups them by expert, executes expert kernels, and restores token order
- Separates total parameters, active parameters, arithmetic, memory, latency, and model quality
- Requires explicit load-balance, overflow, dispatch, precision, and device-placement policies
- May use expert parallelism with all-to-all-style communication across workers
Common Use Cases
- Increasing model parameter capacity under a bounded per-token expert path
- Training sparse language or multimodal models across distributed accelerators
- Studying conditional computation and learned routing
- Combining shared experts with fine-grained routed experts in disclosed architectures
- Serving qualified sparse checkpoints under measured memory, latency, quality, and cost objectives
Example
Loading code...Frequently Asked Questions
What is Mixture of Experts?
Mixture of Experts is a conditional-computation architecture. A router assigns each input to selected expert subnetworks and combines their outputs. In Transformer LLMs, experts are commonly feed-forward modules inside some layers rather than independent end-to-end models.
Does MoE inference cost scale only with active parameters?
No. Active parameters describe the selected weight path but omit shared layers, routing, token permutation, memory traffic, communication, padding, synchronization, and kernel efficiency. Measure end-to-end latency, throughput, memory, quality, and cost on the target system.
Do MoE experts represent distinct knowledge domains?
Not by definition. Experts can develop different routing distributions or internal features, but they may also overlap or specialize in patterns that are not human-readable. Domain labels require routing analysis and controlled quality evidence.
What is expert parallelism?
Expert parallelism places different experts on different workers. Token states are dispatched to workers that own selected experts, processed there, and returned for combination. This saves per-device expert-weight memory but adds communication and load-balance constraints.
How should MoE and dense models be compared?
Fix the checkpoint task, data, quality gates, precision, runtime, hardware topology, batching, concurrency, and sequence distributions. Then report total and active parameters, quality by slice, prefill and decode performance, latency percentiles, memory, communication, failures, and cost.