Direct answer

Mamba is a neural sequence architecture built around a selective state space model: it compresses prior inputs into a recurrent state, but makes the write, retain, and read behavior depend on the current token. This gives its core sequence mixer linear work in sequence length and a fixed-size history state during autoregressive decoding. It does not make Mamba universally faster, lossless, or superior to a Transformer; quality and wall-clock results depend on the exact architecture, state size, kernels, workload, and hardware.

The durable engineering question is therefore not "Will Mamba replace attention?" It is: which information-routing mechanism meets a workload's quality, latency, memory, and operational constraints?

The comparison has three different execution paths

Architecture claims become misleading when training, prompt processing, and token-by-token decoding are merged into one benchmark.

Path Dense causal attention Selective SSM
Training or full-sequence forward pass Common implementations evaluate token interactions in parallel; dense attention has quadratic sequence work Uses a parallel selective scan with linear sequence work
Prompt processing (prefill) Processes all prompt tokens and creates a KV cache Scans the prompt and produces the recurrent state needed for decoding
Autoregressive decode Each new token reads prior KV state, whose history dimension grows with retained tokens Each new token updates a fixed-shape recurrent state

For a sequence of length (L), dense attention materializes or tiles pairwise interactions conceptually associated with an (L \times L) score matrix. IO-aware kernels can avoid storing the full matrix in high-bandwidth memory, but they do not turn dense attention into a linear-time algorithm. During decode, attention's work per new token and KV state both depend on retained history.

A selective SSM performs a recurrence over (L) steps. Its retained history state does not add a new slot for every previous token. However, its total inference memory still includes weights, per-layer recurrent and convolutional state, batch replicas, activations, workspaces, graph reservations, and allocator headroom.

A state space model is a learned recurrence

A State Space Model represents a changing system with a latent state. A common continuous-time form is:

text 
dh(t) / dt = A h(t) + B x(t)
y(t)      = C h(t) + D x(t)

The input (x(t)) changes the latent state (h(t)); the state transition (A) carries information forward; and (C) reads an output. Token sequences are discrete, so an implementation applies a discretization rule:

text 
h_t = A_bar h_(t-1) + B_bar x_t
y_t = C h_t + D x_t

For example, zero-order hold uses a matrix exponential to derive (\bar A) and (\bar B) from continuous parameters and a step size. Other parameterizations can learn discrete dynamics directly. The discretization rule is part of the model, not a cosmetic conversion: it changes stability, expressivity, and numerical behavior.

Parallel training does not remove recurrence

The recurrence defines causal dependencies, but an associative scan can evaluate many recurrence segments in parallel and then combine them. Earlier linear time-invariant SSMs could also be represented as a convolution because their parameters were fixed across positions. Mamba makes parameters input-dependent, so it uses a hardware-aware selective scan rather than relying on one fixed convolution kernel.

flowchart LR X["Input sequence x_1 ... x_L"] --> P["Project token-dependent B, C, and delta"] P --> S["Parallel selective scan for training or prefill"] S --> Y["Sequence outputs y_1 ... y_L"] S --> H["Final recurrent state"] H --> D["One-state update per decode token"]

The algorithm fuses operations, keeps expanded state in faster memory where possible, and recomputes selected intermediates during backward propagation. Whether that wins in wall-clock time depends on the installed kernel, tensor shape, dtype, GPU, and competing attention implementation.

Mamba adds content-dependent selection

Traditional linear time-invariant SSMs apply the same dynamics at every position. That supports efficient convolutional forms but limits content-dependent routing on discrete, information-dense sequences.

The original Mamba paper makes (B), (C), and the discretization step (\Delta) functions of the input. In simplified form:

text 
B_t     = projection_B(x_t)
C_t     = projection_C(x_t)
delta_t = softplus(projection_delta(x_t))

h_t = exp(delta_t A) h_(t-1) + input_term(B_t, delta_t, x_t)
y_t = C_t h_t

This mechanism can vary how strongly a token updates state, how quickly prior state decays, and how the state is read. It is content-dependent, but it is not softmax attention: no query directly indexes an explicit list of all previous value vectors.

A runnable scalar model clarifies the mechanism

The following dependency-free example is not a Mamba implementation. It isolates the selective recurrence: each token supplies a retention coefficient, an input coefficient, and a read coefficient.

python 
from __future__ import annotations

from dataclasses import dataclass
from math import isfinite
from typing import Iterable


@dataclass(frozen=True)
class SelectiveStep:
    value: float
    retain: float
    write: float
    read: float


def selective_recurrence(steps: Iterable[SelectiveStep]) -> list[float]:
    state = 0.0
    outputs: list[float] = []

    for index, step in enumerate(steps):
        values = (step.value, step.retain, step.write, step.read)
        if not all(isfinite(value) for value in values):
            raise ValueError(f"step {index} contains a non-finite value")
        if not 0.0 <= step.retain <= 1.0:
            raise ValueError(f"step {index} retain must be in [0, 1]")

        state = step.retain * state + step.write * step.value
        outputs.append(step.read * state)

    return outputs


sequence = [
    SelectiveStep(value=2.0, retain=0.10, write=1.00, read=1.0),
    SelectiveStep(value=9.0, retain=0.95, write=0.00, read=1.0),
    SelectiveStep(value=1.0, retain=0.50, write=1.00, read=1.0),
]

try:
    print([round(value, 2) for value in selective_recurrence(sequence)])
except ValueError as error:
    raise SystemExit(f"Invalid selective recurrence: {error}") from error

# [2.0, 1.9, 1.95]

The second input has a large numeric value but a zero write coefficient, so it does not enter state. Real Mamba layers learn vector-valued, channel-wise dynamics and combine the SSM with projections, local convolution, gating, normalization, and residual paths.

Mamba-1, Mamba-2, and Mamba-3 solve different bottlenecks

The three generations should not be compressed into a single "faster Mamba" story.

Generation Main mechanism What the source establishes Boundary
Mamba-1 (2023) Input-dependent selective SSM plus hardware-aware scan The paper reports linear sequence scaling, up to 3x selective-scan speed versus compared methods, and 5x generation throughput versus similarly sized Transformers in its setup Paper checkpoints, kernels, A100 tests, tasks, and model sizes; not a universal service multiplier
Mamba-2 (2024) Structured State Space Duality (SSD) and a scalar-identity transition structure The paper reports a 2-8x faster core layer than Mamba-1 while remaining competitive in its language-modeling experiments A layer result is not a 2-8x end-to-end training or serving guarantee
Mamba-3 (2026) Exponential-trapezoidal discretization, complex state, and MIMO At 1.5B scale, the paper reports stronger retrieval/state tracking and Mamba-3 MIMO state size 64 matching Mamba-2 state size 128 perplexity in its experiments A quality-latency Pareto result under stated models and kernels; not "half the memory" for every deployment

What SSD actually says

Mamba-2 connects a structured SSM class to semiseparable matrices and a structured masked-attention form. With a scalar-times-identity transition, the recurrence can be expressed as structured matrix operations that map well to matrix-multiplication hardware.

The safe conclusion is:

A particular structured SSM admits both recurrent and structured masked-matrix views.

The unsafe conclusion is:

Every SSM is mathematically equivalent to standard softmax attention.

Standard attention and an SSD layer differ in parameterization, mask structure, normalization, retained state, and information-routing behavior. The duality is valuable precisely because its assumptions expose where efficient algorithms are possible.

Why Mamba-3 returns to SSM principles

The Mamba-3 paper starts from an inference problem: linear arithmetic complexity can still produce low hardware utilization when decode is memory-bound.

Its three core changes are:

  1. Exponential-trapezoidal discretization creates a more expressive recurrence and an implicit local convolution effect.
  2. Complex-valued state updates act like a data-dependent rotation and improve tested state-tracking capabilities.
  3. Multi-input multi-output (MIMO) state updates replace an outer-product-style update with matrix multiplication, increasing useful arithmetic at similar tested decode latency.

Mamba-3 was submitted in March 2026.

Fixed state is compression, not random access

A fixed-shape recurrent state is a lossy capacity constraint unless a task's sufficient information fits the learned state and update dynamics. It avoids retaining one explicit KV entry per historical token, but it also lacks attention's direct content-addressable path to every retained token representation.

This creates two separate questions:

  • State tracking: can the recurrence update a latent variable correctly over many steps?
  • Retrieval or exact recall: can the model recover a specific earlier fact when queried later?

Neither follows from asymptotic complexity. The Mamba papers use synthetic copying, induction, retrieval, and state-tracking tests because ordinary perplexity can hide these failures. Mamba-3 improves tested state tracking with complex updates, while its retrieval experiments also show that adding a small number of attention layers can improve retrieval quality.

Do not infer usable million-token recall from an (O(L)) complexity statement. Validate the trained checkpoint at the target lengths with distractors, position sweeps, paraphrased queries, and business-task quality measures.

Hybrid attention and SSM is an engineering option

A hybrid can use recurrent layers for fixed-state sequence processing and occasional attention layers for explicit token interaction. The ratio and placement are model design choices, not a universal recipe.

Workload signal Pure attention may fit Pure SSM may fit Hybrid deserves testing
Exact retrieval from arbitrary prior positions Strong candidate Requires direct evidence Strong candidate
Streaming with bounded history state KV policy must be managed Natural recurrent path Strong candidate
Mature serving stack is mandatory Broad ecosystem Check engine and kernel support Check every layer path
Long prompts dominate cost Benchmark optimized prefill Benchmark selective scan Compare prefill and decode separately
State tracking is safety-critical Test explicitly Test explicitly Test explicitly

Hybrid models still retain KV state for their attention layers. Their memory grows more slowly than an otherwise comparable all-attention design only under a specific layer mix and implementation; it does not become constant.

Benchmark the full model, not the architecture label

An architecture comparison is valid only when it records the variables that can explain the result.

json 
{
  "model": {
    "checkpoint": "immutable-model-revision",
    "architecture": "attention|ssm|hybrid",
    "parameters": 2700000000,
    "stateSize": 128,
    "attentionLayerCount": 0
  },
  "runtime": {
    "engine": "name-and-version",
    "kernelRevision": "immutable-revision",
    "dtype": "bf16",
    "quantization": "none",
    "device": "exact-accelerator",
    "deviceCount": 1
  },
  "workload": {
    "promptLengthDistribution": [512, 4096, 16384],
    "outputLength": 256,
    "batchPolicy": "fixed-or-continuous",
    "concurrency": 16
  },
  "metrics": [
    "quality",
    "timeToFirstToken",
    "interTokenLatency",
    "requestThroughput",
    "peakAllocatedMemory",
    "energyOrCost"
  ]
}

Measure at least:

  1. Quality: perplexity plus task metrics, exact recall, state tracking, and failure slices.
  2. Prefill: latency and throughput across the real prompt-length distribution.
  3. Decode: inter-token latency and output throughput at representative concurrency.
  4. Memory: weights, recurrent state or KV cache, activations, workspaces, and fragmentation.
  5. Operations: kernel availability, compilation, precision stability, cancellation, batching, observability, and fallback behavior.

The official Mamba repository provides Mamba-1, Mamba-2, and Mamba-3 modules and benchmark entry points. Its current installation modes distinguish the core package from the opt-in CUDA selective-scan extension. Pin the repository, PyTorch, CUDA, checkpoint, and kernel revisions before treating a result as reproducible.

Production decision checklist

Use this sequence when evaluating an SSM deployment:

  1. Define the information contract. Specify required recall, state tracking, context length, and acceptable quality loss.
  2. Separate execution phases. Benchmark training, prefill, and decode independently before combining end-to-end results.
  3. Hold the comparison constant. Match training tokens, checkpoint quality, precision, hardware, batch policy, and serving features.
  4. Test length generalization. Include lengths beyond training, but do not treat successful execution as successful recall.
  5. Inspect numerical stability. Recurrent dynamics can be precision-sensitive; run long-horizon and perturbation tests.
  6. Validate the engine path. A model architecture is not deployable until the target runtime supports its kernels, batching, monitoring, and failure recovery.
  7. Keep a rollback baseline. Preserve a quality-qualified attention or prior-model route until the new checkpoint meets the same SLO and safety gates.

Common questions

Will Mamba replace Transformers?

No evidence supports a universal replacement claim. Mamba establishes a strong linear-time sequence-model family, while attention retains useful content-addressable interaction and a mature ecosystem. Pure and hybrid designs should compete on a defined workload.

Is Mamba's recurrent state a form of memory?

It is model state, not durable application memory. It carries learned sequence information during a forward or generation session. It does not provide source attribution, cross-session persistence, access control, or reliable fact retrieval by itself.

Can Mamba process a million tokens?

The original paper reports experiments whose performance improved up to million-length sequences in audio and genomics and synthetic extrapolation beyond one million. That does not establish that every language checkpoint has a one-million-token context contract or accurate recall at that length.

Should teams use the paper's 5x or 2-8x numbers for capacity planning?

No. Use them as paper results tied to their compared models and kernels. Capacity planning requires measurements from the exact full model, runtime, device, precision, prompt/output distribution, batch policy, and quality threshold.

Primary sources