Recurrent Neural Network
A recurrent neural network is a class of artificial neural networks that processes sequential data by maintaining an internal hidden state across discrete time steps through cyclical feedback connections.
Recurrent Neural Network (commonly abbreviated as RNN) is a class of artificial neural network in computation and algorithms engineered to process variable-length sequential inputs by maintaining an internal hidden state vector across discrete temporal intervals. While standard feedforward networks operate under the assumption that all input instances are conditionally independent, sequential domainsโsuch as natural language processing, continuous time series forecasting, financial modeling, and algorithmic genomicsโexhibit strong temporal dependencies where current events depend fundamentally on preceding context. A recurrent architecture addresses this challenge by incorporating cyclic feedback connections, enabling information from previous processing steps to persist and modulate subsequent computational states.
The fundamental operational mechanism of a recurrent network relies on parameter sharing across the temporal dimension. By applying an identical set of weight matrices and transformation functions at every discrete time step, the network can process sequences of arbitrary length without scaling its parameter count proportionally to sequence duration. However, training recurrent architectures across extended sequences exposes numerical difficulties, most notably vanishing and diverging gradient signals during gradient-based optimization. As synthesized in the CRV Science Labs Neural Networks Synthesis Review, resolving these dynamical instabilities drove the invention of gated architectures like Long Short-Term Memory and paved the way for modern attention-based transformer models.
Temporal Dynamics and Hidden State Formulations
A standard recurrent network processes an ordered temporal sequence X = (x1, x2, ..., xT), where each xt โ Rd denotes an input feature vector at time step t. The network maintains a continuous-valued hidden state vector ht โ Rm that acts as an internal memory of historical context up to step t.
At each discrete temporal increment, the hidden state updates through a nonlinear recurrence relation combining the current input xt with the previous hidden state htโ1:
ht = ฯh( Whh htโ1 + Wxh xt + bh )
where Wxh โ Rm ร d represents the input-to-hidden weight matrix, Whh โ Rm ร m denotes the recurrent hidden-to-hidden transition matrix, bh โ Rm is an additive bias vector, and ฯh denotes a point-wise nonlinear activation function (typically the hyperbolic tangent function tanh or logistic sigmoid).
When the task requires predictions at each time step (such as token classification) or at sequence termination (such as sequence classification), the hidden state maps to an output prediction vector yฬt:
yฬt = ฯy( Why ht + by )
where Why โ Rk ร m represents the hidden-to-output matrix, by โ Rk denotes the output bias, and ฯy corresponds to a task-appropriate normalization (such as softmax for categorical distributions).
Through computational graph unrolling, a recurrent network unfolding over T time steps is equivalent to a feedforward network possessing T successive layers, with the constraint that weight matrices Wxh, Whh, and Why are tied identically across all layers.
Backpropagation Through Time and Optimization Challenges
Training recurrent architectures employs an adaptation of backpropagation termed Backpropagation Through Time (BPTT). Given a scalar loss function L summing point losses Lt across temporal intervals:
L = โt=1T Lt( yt, yฬt )
The gradient of total loss with respect to the recurrent weight matrix Whh evaluates as:
โL / โWhh = โt=1T โk=1t ( โLt / โht ) ( โht / โhk ) ( โhk / โWhh )
The core mathematical challenge resides in the temporal Jacobian product chain:
โht / โhk = โj=k+1t ( โhj / โhjโ1 ) = โj=k+1t WhhT diag( ฯโฒh( zj ) )
As analyzed theoretically by Bengio, Simard, and Frasconi (1994), this product of t โ k matrices exhibits exponential decay or divergence governed by the spectral radius of the recurrent weight matrix:
- Vanishing Gradients: If the largest singular value of Whh satisfies โWhhโ < 1, and the derivative of the activation function is bounded (such as |tanhโฒ(z)| โค 1), the norm of the gradient decay satisfies โโht / โhkโ โค exp(โฮณ (t โ k)) for positive constant ฮณ. The supervisory error signal decays toward zero across long temporal separations, preventing gradient descent from associating distant temporal events.
- Exploding Gradients: If the spectral radius exceeds unity (โWhhโ > 1), repeated matrix multiplications cause the gradient norm to grow unbounded as exp(ฮณ (t โ k)), inducing severe numerical overflow and destabilizing weight updates.
To mitigate exploding gradients, practitioners implement gradient norm clipping, rescaling gradient vector g whenever its Euclidean norm exceeds threshold c: g โ c g / max(c, โgโ).
Gated Architectures and Constant Error Carousels
To resolve vanishing gradients in deep recurrent sequences, researchers developed specialized gated architectures that decouple memory preservation from continuous state transformation.
Long Short-Term Memory Networks
Introduced by Sepp Hochreiter and Jรผrgen Schmidhuber in 1997, the Long Short-Term Memory (LSTM) architecture replaces standard hidden units with memory cells governed by a Constant Error Carousel (CEC). By maintaining an additive linear update path along a dedicated cell state ct, the derivative โct / โctโ1 retains unity gain, enabling error gradients to propagate across hundreds of time steps without exponential decay or attenuation.
Modern standard LSTMs control memory access through three multiplicative gates:
- Forget Gate (ft): Controls which historical information to discard from the cell state:
ft = ฯ( Wf xt + Uf htโ1 + bf )
- Input Gate (it) and Candidate State (cฬt): Determine which new information to store:
it = ฯ( Wi xt + Ui htโ1 + bi )
cฬt = tanh( Wc xt + Uc htโ1 + bc )
- Cell State Update (ct): Combines retained memory with modulated candidate input via element-wise multiplication (Hadamard product โ):
ct = ft โ ctโ1 + it โ cฬt
- Output Gate (ot) and Hidden State (ht): Modulate which memory elements are emitted into the observable hidden representation:
ot = ฯ( Wo xt + Uo htโ1 + bo )
ht = ot โ tanh( ct )
Through this additive formulation, when ft approaches 1 and it approaches 0, the cell state preserves its contents unchanged indefinitely, preserving information over extended temporal horizons.
Gated Recurrent Units
In 2014, Cho and colleagues proposed the Gated Recurrent Unit (GRU) as a streamlined alternative to the LSTM. The GRU merges the cell state and hidden state into a single vector ht and eliminates the separate output gate, operating with only two gates:
- Update Gate (zt): Determines the balance between retaining the previous hidden state ht and adopting a candidate state.
- Reset Gate (rt): Dictates how much of the previous state is exposed when computing the candidate activation.
By operating with fewer parameter tensors, GRUs achieve computational efficiency while offering empirical performance comparable to LSTMs across many sequence tasks.
Sequence Modeling Transitions: From Recurrence to Attention
Recurrent networks expanded sequence modeling capabilities through specialized topological variants:
- Bidirectional RNNs: Process sequences using two parallel recurrent layers operating in forward and reverse chronological directions, allowing the representation at step t to encode both past context and future context.
- Sequence-to-Sequence (Seq2Seq): An encoder network compresses an input sequence into a fixed-dimensional context vector hT, which a decoder network unpacks into an output sequence of variable length.
Despite their expressive flexibility, recurrent architectures face a structural limitation: the sequential computation of ht depends strictly on htโ1, precluding parallelization across sequence tokens during training. Furthermore, compressing extended context histories into a fixed-width vector introduces an information bottleneck that degrades performance on long-range dependencies.
These structural constraints spurred the development of attention mechanisms, which allow models to compute direct pairwise affinity scores between arbitrary positions in a sequence. The subsequent invention of the Transformer model replaced recurrent loops with parallelized self-attention and residual skip pathways reminiscent of residual neural networks, shifting the sequence processing paradigm from temporal recurrence to dense associative retrieval.
References
- Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation. https://doi.org/10.1162/neco.1997.9.8.1735
- Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks. https://doi.org/10.1109/72.279181
- Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature. https://doi.org/10.1038/323533a0
- LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature. https://doi.org/10.1038/nature14539