📝 Recurrent Neural Network (RNN)

Description

< What is it? >

A Recurrent Neural Network (RNN) is a neural network for ordered data such as text, speech, sensor readings, or time series. It reads one step at a time and carries a hidden state forward, so later steps can use information from earlier ones.

x₁ → [RNN cell] → h₁ → [RNN cell] → h₂ → [RNN cell] → h₃
                  ↑                  ↑                  ↑
                 x₂                 x₃                 x₄

The same RNN cell and its weights are reused at every step. This page uses vanilla RNN to mean the simple Elman-style RNN; LSTM is a more capable gated RNN.

Key points

< Hidden state and recurrence >

At time step $t$ , the RNN combines the current input $x_t$ with its previous hidden state $h_{t-1}$ :

h_t = \tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h)

The hidden state is a learned summary of the sequence seen so far. The matrix $W_{xh}$ processes the current input, while $W_{hh}$ carries information from the previous time step. Both matrices are shared across every step of the sequence.

Example: while reading “The animal rested because it was tired,” the state at it can contain information about the earlier word animal.

< Unrolling through time >

An RNN is easiest to visualize as the same cell copied across time:

x₁, h₀ → cell → h₁
x₂, h₁ → cell → h₂
x₃, h₂ → cell → h₃

The copies are not separate models: they all use the same parameters. This weight sharing lets an RNN accept different sequence lengths without adding new parameters for each position.

< Common output patterns >

Pattern	Input	Output	Example
Many-to-one	A sequence	One prediction after the final step	Predict sentiment for a review
Many-to-many, aligned	A sequence	One prediction per input step	Tag each word for NER
Many-to-many, shifted	A sequence	A next-step prediction at every step	Predict the next token in language modeling

< Training and limitations >

RNNs are trained with backpropagation through time (BPTT): the network is unrolled over its sequence steps, and gradients flow backward through those steps.

This repeated recurrence is the RNN's main weakness. Gradients can vanish or explode over long sequences, making distant information difficult to learn. RNNs also process steps sequentially, so they cannot parallelize across time as easily as Transformers.

LSTM and GRU cells add gates that make long-range learning more reliable. For modern large-scale language modeling, Transformers have largely replaced vanilla RNNs.

< In PyTorch >

import torch
import torch.nn as nn

# Input shape: (batch, time_steps, input_size)
x = torch.randn(32, 20, 128)

rnn = nn.RNN(
    input_size=128,
    hidden_size=256,
    batch_first=True,
)

output, h_n = rnn(x)

# output: one hidden state per time step, shape (32, 20, 256)
# h_n: final hidden state, shape (1, 32, 256)

Comparison

< Vanilla RNN vs LSTM >

	Vanilla RNN	LSTM
Memory	One hidden state $h_t$	Hidden state plus a cell state $c_t$
Controls	No explicit memory gates	Forget, input, and output gates
Long dependencies	Often difficult because gradients vanish or explode	More reliable, though not perfect
Compute per step	Lower	Higher because it computes several gates
Typical use today	Small or simple sequential baselines	Time series, speech, and legacy sequence models

LSTM extends an RNN with gated memory.
Embeddings provide numeric token vectors that sequence models can consume.
Named Entity Recognition is one many-to-many sequence-labeling task.
Vanishing & Exploding Gradients explains the main training difficulty of vanilla RNNs.
Transformer is the attention-based sequence architecture that now dominates many NLP tasks.

Description​

< What is it? >​

Key points​

< Hidden state and recurrence >​

< Unrolling through time >​

< Common output patterns >​

< Training and limitations >​

< In PyTorch >​

Comparison​

< Vanilla RNN vs LSTM >​

Related ideas​

Reference​