What is Mixture of Experts?
Mixture of Experts (MoE) is a conditional-computation architecture in which a learned or fixed routing policy selects a subset of expert subnetworks for each input and combines their outputs.
Quick Facts
| Full Name | Mixture of Experts (MoE) |
|---|---|
| Created | 1991 by Jacobs et al., popularized in LLMs since 2022 |
| Specification | Official Specification |
How It Works
In many Transformer language models, an MoE block replaces selected dense feed-forward layers with a router and a pool of expert FFNs. The router scores token states, selects one or more experts, dispatches token states to them, and combines their weighted outputs. Sparse activation can increase total parameter capacity without executing every expert for every token, but it does not make total parameters, active parameters, FLOPs, latency, memory, or quality interchangeable. Real systems must store or shard expert weights, balance loads, group token assignments into efficient kernels, and communicate across devices. Experts may develop statistical specialization, but the architecture does not guarantee human-readable domains. See the <a href="https://qubittool.com/blog/moe-architecture-explained">MoE architecture guide</a> for training, serving, and benchmark contracts.
Key Characteristics
- Uses conditional computation to activate a subset of subnetworks for each input
- Commonly replaces selected Transformer FFN blocks while other components remain dense or shared
- Routes batched token states, groups them by expert, executes expert kernels, and restores token order
- Separates total parameters, active parameters, arithmetic, memory, latency, and model quality
- Requires explicit load-balance, overflow, dispatch, precision, and device-placement policies
- May use expert parallelism with all-to-all-style communication across workers
Common Use Cases
- Increasing model parameter capacity under a bounded per-token expert path
- Training sparse language or multimodal models across distributed accelerators
- Studying conditional computation and learned routing
- Combining shared experts with fine-grained routed experts in disclosed architectures
- Serving qualified sparse checkpoints under measured memory, latency, quality, and cost objectives
Example
Loading code...Frequently Asked Questions
What is Mixture of Experts?
Mixture of Experts is a conditional-computation architecture. A router assigns each input to selected expert subnetworks and combines their outputs. In Transformer LLMs, experts are commonly feed-forward modules inside some layers rather than independent end-to-end models.
Does MoE inference cost scale only with active parameters?
No. Active parameters describe the selected weight path but omit shared layers, routing, token permutation, memory traffic, communication, padding, synchronization, and kernel efficiency. Measure end-to-end latency, throughput, memory, quality, and cost on the target system.
Do MoE experts represent distinct knowledge domains?
Not by definition. Experts can develop different routing distributions or internal features, but they may also overlap or specialize in patterns that are not human-readable. Domain labels require routing analysis and controlled quality evidence.
What is expert parallelism?
Expert parallelism places different experts on different workers. Token states are dispatched to workers that own selected experts, processed there, and returned for combination. This saves per-device expert-weight memory but adds communication and load-balance constraints.
How should MoE and dense models be compared?
Fix the checkpoint task, data, quality gates, precision, runtime, hardware topology, batching, concurrency, and sequence distributions. Then report total and active parameters, quality by slice, prefill and decode performance, latency percentiles, memory, communication, failures, and cost.