โ† Wiki Index / Computation & Algorithms / Recurrent Neural Network
๐Ÿ’ป Computation & Algorithms

Recurrent Neural Network

A recurrent neural network is a class of artificial neural networks that processes sequential data by maintaining an internal hidden state across discrete time steps through cyclical feedback connections.

Recurrent Neural Network (commonly abbreviated as RNN) is a class of artificial neural network in computation and algorithms engineered to process variable-length sequential inputs by maintaining an internal hidden state vector across discrete temporal intervals. While standard feedforward networks operate under the assumption that all input instances are conditionally independent, sequential domainsโ€”such as natural language processing, continuous time series forecasting, financial modeling, and algorithmic genomicsโ€”exhibit strong temporal dependencies where current events depend fundamentally on preceding context. A recurrent architecture addresses this challenge by incorporating cyclic feedback connections, enabling information from previous processing steps to persist and modulate subsequent computational states.

The fundamental operational mechanism of a recurrent network relies on parameter sharing across the temporal dimension. By applying an identical set of weight matrices and transformation functions at every discrete time step, the network can process sequences of arbitrary length without scaling its parameter count proportionally to sequence duration. However, training recurrent architectures across extended sequences exposes numerical difficulties, most notably vanishing and diverging gradient signals during gradient-based optimization. As synthesized in the CRV Science Labs Neural Networks Synthesis Review, resolving these dynamical instabilities drove the invention of gated architectures like Long Short-Term Memory and paved the way for modern attention-based transformer models.

Temporal Dynamics and Hidden State Formulations

A standard recurrent network processes an ordered temporal sequence X = (x1, x2, ..., xT), where each xt โˆˆ Rd denotes an input feature vector at time step t. The network maintains a continuous-valued hidden state vector ht โˆˆ Rm that acts as an internal memory of historical context up to step t.

At each discrete temporal increment, the hidden state updates through a nonlinear recurrence relation combining the current input xt with the previous hidden state htโˆ’1:

ht = ฯƒh( Whh htโˆ’1 + Wxh xt + bh )

where Wxh โˆˆ Rm ร— d represents the input-to-hidden weight matrix, Whh โˆˆ Rm ร— m denotes the recurrent hidden-to-hidden transition matrix, bh โˆˆ Rm is an additive bias vector, and ฯƒh denotes a point-wise nonlinear activation function (typically the hyperbolic tangent function tanh or logistic sigmoid).

When the task requires predictions at each time step (such as token classification) or at sequence termination (such as sequence classification), the hidden state maps to an output prediction vector yฬ‚t:

yฬ‚t = ฯƒy( Why ht + by )

where Why โˆˆ Rk ร— m represents the hidden-to-output matrix, by โˆˆ Rk denotes the output bias, and ฯƒy corresponds to a task-appropriate normalization (such as softmax for categorical distributions).

Through computational graph unrolling, a recurrent network unfolding over T time steps is equivalent to a feedforward network possessing T successive layers, with the constraint that weight matrices Wxh, Whh, and Why are tied identically across all layers.

Backpropagation Through Time and Optimization Challenges

Training recurrent architectures employs an adaptation of backpropagation termed Backpropagation Through Time (BPTT). Given a scalar loss function L summing point losses Lt across temporal intervals:

L = โˆ‘t=1T Lt( yt, yฬ‚t )

The gradient of total loss with respect to the recurrent weight matrix Whh evaluates as:

โˆ‚L / โˆ‚Whh = โˆ‘t=1T โˆ‘k=1t ( โˆ‚Lt / โˆ‚ht ) ( โˆ‚ht / โˆ‚hk ) ( โˆ‚hk / โˆ‚Whh )

The core mathematical challenge resides in the temporal Jacobian product chain:

โˆ‚ht / โˆ‚hk = โˆj=k+1t ( โˆ‚hj / โˆ‚hjโˆ’1 ) = โˆj=k+1t WhhT diag( ฯƒโ€ฒh( zj ) )

As analyzed theoretically by Bengio, Simard, and Frasconi (1994), this product of t โˆ’ k matrices exhibits exponential decay or divergence governed by the spectral radius of the recurrent weight matrix:

  • Vanishing Gradients: If the largest singular value of Whh satisfies โ€–Whhโ€– < 1, and the derivative of the activation function is bounded (such as |tanhโ€ฒ(z)| โ‰ค 1), the norm of the gradient decay satisfies โ€–โˆ‚ht / โˆ‚hkโ€– โ‰ค exp(โˆ’ฮณ (t โˆ’ k)) for positive constant ฮณ. The supervisory error signal decays toward zero across long temporal separations, preventing gradient descent from associating distant temporal events.
  • Exploding Gradients: If the spectral radius exceeds unity (โ€–Whhโ€– > 1), repeated matrix multiplications cause the gradient norm to grow unbounded as exp(ฮณ (t โˆ’ k)), inducing severe numerical overflow and destabilizing weight updates.

To mitigate exploding gradients, practitioners implement gradient norm clipping, rescaling gradient vector g whenever its Euclidean norm exceeds threshold c: g โ† c g / max(c, โ€–gโ€–).

Gated Architectures and Constant Error Carousels

To resolve vanishing gradients in deep recurrent sequences, researchers developed specialized gated architectures that decouple memory preservation from continuous state transformation.

Long Short-Term Memory Networks

Introduced by Sepp Hochreiter and Jรผrgen Schmidhuber in 1997, the Long Short-Term Memory (LSTM) architecture replaces standard hidden units with memory cells governed by a Constant Error Carousel (CEC). By maintaining an additive linear update path along a dedicated cell state ct, the derivative โˆ‚ct / โˆ‚ctโˆ’1 retains unity gain, enabling error gradients to propagate across hundreds of time steps without exponential decay or attenuation.

Modern standard LSTMs control memory access through three multiplicative gates:

  1. Forget Gate (ft): Controls which historical information to discard from the cell state:

ft = ฯƒ( Wf xt + Uf htโˆ’1 + bf )

  1. Input Gate (it) and Candidate State (cฬƒt): Determine which new information to store:

it = ฯƒ( Wi xt + Ui htโˆ’1 + bi )

cฬƒt = tanh( Wc xt + Uc htโˆ’1 + bc )

  1. Cell State Update (ct): Combines retained memory with modulated candidate input via element-wise multiplication (Hadamard product โŠ™):

ct = ft โŠ™ ctโˆ’1 + it โŠ™ cฬƒt

  1. Output Gate (ot) and Hidden State (ht): Modulate which memory elements are emitted into the observable hidden representation:

ot = ฯƒ( Wo xt + Uo htโˆ’1 + bo )

ht = ot โŠ™ tanh( ct )

Through this additive formulation, when ft approaches 1 and it approaches 0, the cell state preserves its contents unchanged indefinitely, preserving information over extended temporal horizons.

Gated Recurrent Units

In 2014, Cho and colleagues proposed the Gated Recurrent Unit (GRU) as a streamlined alternative to the LSTM. The GRU merges the cell state and hidden state into a single vector ht and eliminates the separate output gate, operating with only two gates:

  • Update Gate (zt): Determines the balance between retaining the previous hidden state ht and adopting a candidate state.
  • Reset Gate (rt): Dictates how much of the previous state is exposed when computing the candidate activation.

By operating with fewer parameter tensors, GRUs achieve computational efficiency while offering empirical performance comparable to LSTMs across many sequence tasks.

Sequence Modeling Transitions: From Recurrence to Attention

Recurrent networks expanded sequence modeling capabilities through specialized topological variants:

  • Bidirectional RNNs: Process sequences using two parallel recurrent layers operating in forward and reverse chronological directions, allowing the representation at step t to encode both past context and future context.
  • Sequence-to-Sequence (Seq2Seq): An encoder network compresses an input sequence into a fixed-dimensional context vector hT, which a decoder network unpacks into an output sequence of variable length.

Despite their expressive flexibility, recurrent architectures face a structural limitation: the sequential computation of ht depends strictly on htโˆ’1, precluding parallelization across sequence tokens during training. Furthermore, compressing extended context histories into a fixed-width vector introduces an information bottleneck that degrades performance on long-range dependencies.

These structural constraints spurred the development of attention mechanisms, which allow models to compute direct pairwise affinity scores between arbitrary positions in a sequence. The subsequent invention of the Transformer model replaced recurrent loops with parallelized self-attention and residual skip pathways reminiscent of residual neural networks, shifting the sequence processing paradigm from temporal recurrence to dense associative retrieval.

References

  1. Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation. https://doi.org/10.1162/neco.1997.9.8.1735
  2. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks. https://doi.org/10.1109/72.279181
  3. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature. https://doi.org/10.1038/323533a0
  4. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature. https://doi.org/10.1038/nature14539
๐Ÿงฌ Support Open Science
Explore interactive science games on LowKey Games or browse scientific apparel & accessories in the CRV Science Shop.
๐Ÿ’ก Spot an inaccuracy or have primary literature to expand this article?
Submit a citation or revision note using your shared CRV Science / LowKey Games profile ID.
Suggest Revision