What is State Space Model?
State Space Model (SSM) represents a dynamic system or sequence with a latent state that is updated from the previous state and current input, then mapped to an output. In neural sequence models, this recurrence can provide linear sequence processing and a fixed-shape history state during autoregressive decoding.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
A classical continuous-time state space model uses a state equation dh(t)/dt = A h(t) + B x(t) and an output equation y(t) = C h(t) + D x(t). Sequence models discretize these dynamics into a recurrence such as h_t = A_bar h_(t-1) + B_bar x_t. The discretization rule and parameterization affect stability, expressivity, and numerical behavior. Structured SSMs constrain the transition matrix so the recurrence can be evaluated efficiently as a convolution, scan, or structured matrix operation. Selective SSMs such as Mamba make parts of the dynamics input-dependent, allowing the model to vary how it retains, writes, and reads information. A fixed-shape recurrent state does not mean the entire model uses constant memory, and it does not guarantee exact recall: weights, batch size, layers, state width, activations, workspaces, and kernels still matter.
Key Characteristics
- Represents sequence history with a latent state updated by a recurrence
- Uses a discretization or direct discrete parameterization to process sampled sequences
- Can support parallel full-sequence algorithms and recurrent token-by-token execution
- Structured variants constrain dynamics to make convolution, scan, or matrix algorithms efficient
- Selective variants make state updates or reads depend on the current input
- Fixed-shape history state trades explicit random access for compressed recurrent memory
Common Use Cases
- Neural language, audio, vision, genomic, and time-series sequence models
- Streaming inference where bounded history state is an operational requirement
- Control and signal-processing systems that estimate latent dynamics from observations
- Hybrid neural architectures combining recurrent state-space layers with attention
- Long-sequence experiments comparing quality, state tracking, retrieval, latency, and memory
Example
Loading code...Frequently Asked Questions
What is a state space model in machine learning?
A state space model represents prior sequence information with a latent state. Each step updates that state from the previous state and current input, then reads an output. Neural SSMs learn these dynamics and may use structured algorithms for parallel training or recurrent inference.
How is a state space model different from an RNN?
Both can use recurrent state, but modern structured SSMs derive or parameterize dynamics so they can also use efficient convolution, scan, or structured matrix algorithms. RNN is a broad neural architecture family; SSM identifies a state-dynamics formulation with explicit transition, input, and readout structure.
How is a state space model different from attention?
An SSM updates a compressed latent state as tokens arrive. Attention computes data-dependent interactions over permitted token representations. Some structured SSMs have matrix representations related to structured masked attention, but this does not make every SSM equivalent to standard softmax attention.
Do state space models use constant memory?
Their retained history state during recurrent decoding can have a fixed shape independent of generated sequence length. Total memory is not constant: it still scales with model weights, layers, batch size, state width, precision, activations, workspaces, and implementation details.
What is a selective state space model?
A selective SSM makes parts of its dynamics depend on the current input. In Mamba, input-dependent parameters control how information is written, retained, and read at each position. This adds content-dependent behavior while preserving a recurrent, linear-sequence computation path.