What is RNN?
RNN (Recurrent Neural Network) is a neural-network architecture that processes an ordered sequence by repeatedly applying shared parameters to the current input and previous hidden state, producing a state and optional output at each step.
Quick Facts
| Full Name | Recurrent Neural Network |
|---|---|
| Created | 1986 by David Rumelhart, Geoffrey Hinton, and Ronald Williams |
| Specification | Official Specification |
How It Works
State creates order-sensitive computation
Changing input order changes the chain of hidden states. A causal RNN consumes only the past and can emit online; a bidirectional RNN uses both directions and therefore requires the complete segment. Sequence-to-one, sequence-to-sequence, and per-step heads differ in where loss is applied. Padding masks and sequence boundaries are part of the model contract.
BPTT exposes the training boundary
Backpropagation Through Time unrolls recurrence and accumulates derivatives through repeated state transitions. Long products of Jacobians can shrink or grow, causing vanishing or exploding gradients. Gradient clipping controls explosion but does not restore lost signal. Truncated BPTT limits memory and compute while also limiting how far credit can propagate.
LSTM and GRU mitigate, not eliminate, long-range failures
LSTM adds a cell state with input, forget, and output gates; GRU combines gating into a smaller state update. These paths can preserve gradients and information better than a vanilla RNN, but they do not guarantee retention over arbitrary lengths. Initialization, normalization, truncation, optimization, data, and hidden-state size still matter.
State lifecycle determines production correctness
Reset state at entity or session boundaries, carry it only with an explicit identity and ordering contract, and prevent future information from entering causal evaluation. Compare RNNs with temporal CNNs, Transformers, and state-space models on quality, context length, streaming latency, memory, throughput, missing events, reordered events, and recovery after a gap.
Key Characteristics
- Reuses the same transition parameters across sequence positions
- Maintains a fixed-width hidden state that summarizes prior inputs
- Supports online causal inference one event at a time
- Uses BPTT or truncated BPTT to train recurrent transitions
- Can suffer vanishing, exploding, stale, or cross-session state
- Includes vanilla, bidirectional, LSTM, GRU, and stacked variants
Common Use Cases
- Streaming sensor or audio classification with bounded per-step state
- Low-latency forecasting where observations arrive in strict order
- Sequence labeling when a compact recurrent model meets the quality target
- On-device workloads constrained by memory and batch size
- Stateful anomaly detection with explicit reset and gap handling
Example
Loading code...Frequently Asked Questions
What information does an RNN hidden state contain?
It is a learned, compressed function of previous state and current input. It is not a literal transcript and has no guaranteed retention horizon. What survives depends on the transition dynamics, state width, objective, training sequence lengths, truncation, and data.
What is the difference between an RNN, LSTM, and GRU?
A vanilla RNN applies a simple recurrent state update. LSTM adds a separate cell state and three gates; GRU uses a more compact gated update. LSTM and GRU often preserve useful signals over longer spans, but neither automatically solves long-context reasoning or every gradient problem.
What is truncated backpropagation through time?
Truncated BPTT carries state forward but backpropagates gradients through only a bounded window. It reduces activation memory and update latency, making long streams trainable, while preventing credit assignment beyond the truncation boundary. The window is therefore both a resource setting and a modeling assumption.
When should I choose an RNN instead of a Transformer?
Consider an RNN when inputs arrive one step at a time, state is small, latency must remain bounded, and the required dependency horizon is compatible with recurrence. Use measured comparisons: Transformers offer parallel training and flexible global context, while temporal CNNs or state-space models may provide other latency and memory trade-offs.
How do hidden-state bugs cause data leakage?
Reusing state across unrelated users, devices, files, or evaluation examples leaks prior information and changes predictions. Bidirectional models also use future context and cannot be evaluated as causal streaming systems. Tests should assert reset boundaries, ordering, masking, serialization, gap recovery, and parity between training and serving.