TL;DR

Mixture of Experts (MoE) is a conditional-computation architecture. A router scores each token representation, selects one or more expert subnetworks, dispatches token states to them, and combines their outputs. This can increase total parameter capacity without running every expert for every token.

Sparse activation is not free compute. A production MoE still pays for shared layers, expert-weight storage, routing, token permutation, memory movement, collective communication, uneven expert loads, and many small matrix multiplications. Therefore:

text 
total parameters != active parameters != FLOPs != latency != quality

The useful engineering question is not whether MoE is universally more efficient. It is whether a specific MoE release delivers better accepted quality, latency, throughput, and cost under a fixed workload and hardware contract.

Table of Contents

What MoE Changes in a Transformer

The original 1991 adaptive mixtures of local experts used separate networks and a learned gating network to divide training cases into subtasks. Modern sparse language models apply the same conditional-computation idea at much larger scale.

In a common Transformer design, selected dense feed-forward network (FFN) blocks are replaced by MoE blocks:

text 
token states
-> normalization and attention
-> residual path
-> MoE router
-> selected expert FFNs
-> weighted combination
-> residual path

Attention, embeddings, normalization, output heads, and some FFN layers may remain dense or shared. Architectures also differ in how frequently they insert MoE layers, whether they include always-on shared experts, how many routed experts exist, and how many are selected.

An expert is usually an FFN with its own weights. It is not automatically a human-readable specialist. A router can learn repeatable preferences without producing clean categories such as "Python," "biology," or "grammar." Semantic specialization must be demonstrated with routing statistics, controlled interventions, and quality measurements.

flowchart LR X[Token states] --> R[Router logits] R --> K[Top-k or other sparse selection] K --> P[Permute and dispatch by expert] P --> E[Expert FFN computation] E --> C[Combine weighted outputs] C --> U[Unpermute to token order] U --> Y[Residual output]

The Batched Routing Data Path

A one-token loop hides the hard part of MoE. Real training and serving route a batch of token states:

  1. Score: the router maps each token state to logits over experts.
  2. Select: a routing policy chooses top-k experts, expert groups, or another sparse assignment.
  3. Normalize: selected scores become combination weights according to the architecture.
  4. Account for capacity: the implementation determines whether assignments are padded, dropped, rerouted, or executed with variable sizes.
  5. Permute: token states are grouped by destination expert.
  6. Dispatch: when experts live on other devices, collectives move token states to their owners.
  7. Compute: grouped or batched matrix multiplications run local experts.
  8. Combine and restore: expert outputs are weighted, returned, and restored to original token order.

The conceptual equation for token state x_t is:

text 
MoE(x_t) = sum over selected experts i of g_i(x_t) * E_i(x_t)

That equation omits capacity, device placement, communication, padding, precision, and kernel scheduling. It describes model semantics, not a production implementation.

The 2017 sparsely-gated MoE paper established that a trainable gate could activate a sparse combination of many FFN experts. Switch Transformer then showed one-expert routing as a specific simplification. Neither result makes top-1 or top-2 a universal default.

Total Parameters Active Parameters and Real Cost

Four quantities must be reported separately:

Quantity What it describes What it does not prove
Total parameters All model weights, including inactive experts Per-token arithmetic, latency, or quality
Active parameters Weights used along a token's selected path Actual FLOPs, memory traffic, or dense-model equivalence
FLOPs Arithmetic under a stated sequence and batch shape Kernel utilization, communication, or wall-clock latency
Resident memory Weights, KV cache, activations, workspaces, runtime state Whether the workload meets latency or throughput objectives

Active parameters can be a useful architecture descriptor, but it is not a performance model. Shared attention and dense layers still run. Expert weights must be resident, sharded, streamed, or offloaded. Routing creates data movement and synchronization. Small or uneven expert batches can underuse accelerators.

For one release and workload, reason about:

text 
request_cost =
  dense_shared_compute
  + routed_expert_compute
  + router_and_permutation
  + local_memory_traffic
  + expert_parallel_communication
  + synchronization_and_padding
  + runtime_overhead

This is why "13B active parameters" does not establish "13B dense-model speed," and total parameters do not establish dense-model-equivalent quality.

Training Balance Without Erasing Useful Routing

Routers and experts are trained jointly, but discrete selection and uneven demand create several coupled problems:

  • a few experts may receive most tokens;
  • underused experts may learn slowly;
  • overloaded devices can determine step time;
  • capacity limits can drop assignments or waste padding;
  • a strong balancing objective can distort the language-model objective;
  • low-precision router decisions can change assignments near score boundaries.

Auxiliary load-balancing loss is one design, not a law. Switch Transformer used balancing and capacity mechanisms, while DeepSeek-V3 describes a dynamic expert-bias strategy intended to balance load without a conventional auxiliary loss. Production frameworks expose several alternatives, including auxiliary losses, sequence- or global-level balance, Sinkhorn-style routing, dynamic bias, and no balancing.

Track at least:

text 
tokens per expert and per device
router probability mass per expert
coefficient of variation and peak-to-mean load
overflow, drop, reroute, and padding rates
expert batch-size distribution
router entropy and assignment churn
language-model loss and downstream quality by slice
communication and expert-compute time

Uniform routing is not the final objective. The target is stable quality and efficient execution without dead experts or uncontrolled hotspots.

Serving Memory Communication and Kernels

Expert parallelism is not tensor parallelism

  • Expert parallelism (EP) places different experts on different workers and routes token states to the workers that own selected experts.
  • Tensor parallelism (TP) shards tensors within a layer and synchronizes partial results.
  • Pipeline parallelism (PP) assigns layer ranges to stages.
  • Data parallelism (DP) replicates model execution across data shards.

Large deployments combine these dimensions. The best mapping depends on expert count, hidden sizes, sequence lengths, topology, memory, and traffic. It cannot be selected from total parameter count alone.

Dispatch is a systems operation

With EP, a typical MoE layer performs an all-to-all-style dispatch, local expert computation, and a return/combine collective. PyTorch's large-scale MoE account and Megatron-Core's MoE guide both expose this data movement as a first-class concern.

sequenceDiagram participant A as GPU A tokens participant D as Dispatcher participant B as GPU B experts participant C as Combiner A->>D: token states plus expert assignments D->>B: all-to-all dispatch B->>B: grouped expert GEMMs B->>C: all-to-all return C->>A: weighted outputs in original order

The runtime may use dropless variable-size execution, capacity padding, expert replication, grouped GEMMs, fused permutation kernels, communication overlap, or topology-aware routing. These are implementation choices. The old claim that serving frameworks universally use token dropping to avoid crashes confuses a capacity-based training option with all inference systems.

Local execution needs a measured memory plan

Quantization reduces weight storage, but local feasibility also depends on:

  • exact checkpoint and quantization format;
  • runtime metadata and temporary buffers;
  • KV cache for context length, batch, and concurrency;
  • CPU, GPU, or unified-memory placement;
  • memory bandwidth and offload traffic;
  • prompt processing and decode targets.

No fixed RAM number proves that an MoE model will run comfortably. Publish the checkpoint digest, runtime revision, context, concurrency, token rates, latency percentiles, peak memory, and quality checks.

Prefill and Decode Stress Different Paths

Prefill processes many prompt tokens together. It may create larger expert batches and more efficient grouped GEMMs, but long prompts increase attention, activation, KV-cache, dispatch, and network volume.

Decode usually advances each active sequence by one token per step. Continuous batching can aggregate work, but expert batches may remain smaller and more uneven. Kernel launches, synchronization, memory bandwidth, and interconnect latency can dominate.

Report both paths:

text 
prefill: input-token throughput, time to first token, expert batch sizes
decode: output-token throughput, time per output token, inter-token latency
both: p50/p95/p99 latency, load imbalance, communication share, peak memory

An optimization that improves prefill throughput can leave decode unchanged or worse.

What Published Architectures Actually Establish

Use disclosed architectures as bounded examples:

  • Mixtral 8x7B reports eight FFN experts per layer, two selected per token, 47B total parameters, and 13B active parameters for that checkpoint. These figures do not transfer to every MoE.
  • DeepSeekMoE studies finer-grained routed experts and always-on shared experts to reduce redundancy and improve specialization in its evaluated designs.
  • DeepSeek-V3 reports 671B total and 37B activated parameters, auxiliary-loss-free balancing, node-limited routing, no token dropping in its stated training setup, and separate prefill and decode deployment strategies.
  • Switch Transformer evaluates top-1 routing and capacity controls in its own training configuration.

Closed-model internals should not be inferred from leaks or repeated speculation. OpenAI's public GPT-4 materials do not disclose an expert count or the leaked parameter arithmetic previously shown on this page, so those claims are not evidence for an MoE architecture.

A Reproducible MoE Evaluation Contract

Before comparing dense and sparse candidates, freeze:

yaml 
artifact:
  checkpoint: model/revision
  tokenizer: tokenizer/revision
  precision: bf16
  runtime: runtime/revision

architecture:
  totalParameters: reported
  activeParameters: reported
  moeLayerFrequency: reported
  routedExperts: reported
  selectedExperts: reported
  sharedExperts: reported

workload:
  inputLengthDistribution: dataset/revision
  outputLengthDistribution: dataset/revision
  arrivalProcess: trace/revision
  concurrency: reported
  batchingPolicy: config/revision

system:
  accelerators: type-and-count
  topology: node-and-interconnect-map
  parallelism: dp-tp-pp-ep
  memoryPolicy: residency-sharding-offload

gates:
  quality: task-and-slice-thresholds
  safety: policy-thresholds
  latency: ttft-tpot-end-to-end
  reliability: error-and-oom-budget

report:
  - accepted-output-goodput
  - p50-p95-p99-latency
  - prefill-and-decode-throughput
  - peak-and-resident-memory
  - communication-share
  - expert-load-distribution
  - cost-per-accepted-output

Change one variable at a time or declare the comparison non-causal. A useful benchmark reports rejected outputs and failed requests, not only raw tokens per second.

Failure Modes and Diagnostics

Symptom Evidence to collect Possible causes
A few experts dominate token counts, router mass, entropy, per-slice routes router collapse, genuine skew, weak or excessive balance control
Tail latency rises per-expert queue, per-device load, all-to-all time hot experts, topology crossing, small GEMMs, synchronization
Training quality regresses task slices, routing churn, drop rate, loss terms overflow, unstable routing, balance objective interference
More experts do not help iso-compute quality curves, utilization, memory diminishing capacity returns, insufficient data, communication overhead
Local runtime is slow bandwidth, offload bytes, page faults, peak memory weights do not remain resident, KV-cache pressure, poor kernels
Quantized output changes routing agreement, per-slice quality, calibration router score perturbation, expert-weight error, runtime mismatch

Do not diagnose an MoE only from average utilization. Correlate model quality with routing, expert batches, collectives, memory, and request-level latency.

FAQ

Is MoE always faster than a dense model

No. Sparse expert arithmetic may be lower than a same-total-parameter dense design, but communication, memory traffic, small expert batches, imbalance, and shared layers can dominate. Only an end-to-end benchmark under the target workload answers the question.

Must every MoE route two experts per token

No. Published systems use different top-k values, shared experts, grouped routing, and balancing strategies. Top-k is part of a versioned architecture, not an industry constant.

Are experts separate models

Usually not in a Transformer MoE. They are typically FFN modules inside selected layers and share the rest of the model. Their outputs participate in one forward pass.

Does expert capacity mean inference must drop tokens

No. Capacity and token dropping are implementation choices. Dropless kernels and variable-size expert batches exist, and some architectures explicitly avoid dropping. Record the actual runtime behavior.

Can active parameter count compare model quality

No. Quality depends on training data, optimization, architecture, total capacity, active path, tokenizer, post-training, and evaluation protocol. Compare quality directly on representative slices.

Summary

MoE turns some dense computation into conditional computation:

text 
route
-> group and dispatch
-> execute selected experts
-> combine and restore

Its value comes from increasing model capacity without activating every expert for every token. Its cost appears in weight residency, routing stability, expert load, collective communication, kernel efficiency, and operational complexity. Treat total parameters, active parameters, FLOPs, memory, latency, throughput, cost, and quality as separate measurements.