📝 Long Short-Term Memory (LSTM)

Description

< What is it? >

A Long Short-Term Memory (LSTM) network is a gated type of RNN. It was designed to preserve useful information and gradient flow across longer sequences than a vanilla RNN can usually handle.

An LSTM cell carries two vectors from one time step to the next:

$c_t$ : the cell state, a long-running memory path
$h_t$ : the hidden state, the output exposed to the next layer or time step

Key points

< Gates control memory >

At each time step, an LSTM uses learned sigmoid gates to decide what to forget, write, and expose. Let

z_t = \begin{bmatrix} h_{t-1}\\ x_t \end{bmatrix}

The gates and state updates are:

\begin{aligned} f_t&=\sigma(W_fz_t+b_f) && \text{forget gate}\\ i_t&=\sigma(W_iz_t+b_i) && \text{input gate}\\ \tilde c_t&=\tanh(W_cz_t+b_c) && \text{candidate memory}\\ c_t&=f_t\odot c_{t-1}+i_t\odot\tilde c_t && \text{new cell state}\\ o_t&=\sigma(W_oz_t+b_o) && \text{output gate}\\ h_t&=o_t\odot\tanh(c_t) && \text{new hidden state} \end{aligned}

Here, $\sigma$ gives values between $0$ and $1$ , and $\odot$ means element-wise multiplication.

Gate	Simple interpretation
Forget gate $f_t$	How much of the previous memory to keep
Input gate $i_t$	How much new candidate information to write
Output gate $o_t$	How much of the cell memory to reveal as $h_t$

< Why LSTM helps with long sequences >

The cell-state update has an additive path:

c_t = f_t\odot c_{t-1}+i_t\odot\tilde c_t

When the forget gate is near $1$ , useful memory and its gradient can pass through many time steps with less shrinkage than in a vanilla RNN. Gates can also discard irrelevant information when it is no longer useful.

LSTM reduces the vanishing-gradient problem; it does not make every long dependency easy or guarantee that gradients never explode.

Example: after reading “The animal rested because it was tired,” an LSTM can choose to retain information about animal until it processes it.

< Inputs and outputs >

For a batch-first PyTorch input with shape (B, T, D):

Symbol	Meaning	Typical shape
$B$	Batch size	—
$T$	Number of time steps	—
$D$	Input feature or embedding size	—
$H$	LSTM hidden size	—
Output	Hidden state for every step	`(B, T, H)`
Final states	Final hidden and cell states	`(L, B, H)` each

Here, $L$ is the number of stacked LSTM layers for a one-directional LSTM.

< In PyTorch >

import torch
import torch.nn as nn

# Input shape: (batch, time_steps, embedding_size)
x = torch.randn(32, 20, 128)

lstm = nn.LSTM(
    input_size=128,
    hidden_size=256,
    batch_first=True,
)

output, (h_n, c_n) = lstm(x)

# output: (32, 20, 256), one hidden state per time step
# h_n:    (1, 32, 256), final hidden state
# c_n:    (1, 32, 256), final cell state

< Where it is used >

LSTMs remain useful for modest-size time-series, speech, sensor, and sequence tasks, especially when sequential processing is acceptable. For very long text or large-scale language modeling, Transformers usually provide better parallelism and global-context modeling.

Comparison

< LSTM vs GRU >

	LSTM	GRU
State	Separate hidden state and cell state	One hidden state
Gates	Forget, input, output	Update and reset
Parameters	More	Fewer
Practical choice	Useful when explicit cell memory is helpful	Often a simpler, faster gated baseline

Both are gated RNNs and are usually more capable than a vanilla RNN for long dependencies.

Recurrent Neural Network (RNN) introduces recurrence and hidden states.
Embeddings explains how text tokens become LSTM inputs.
Vanishing & Exploding Gradients explains why gated cells are helpful.
Transformer provides a non-recurrent alternative for sequence modeling.

Description​

< What is it? >​

Key points​

< Gates control memory >​

< Why LSTM helps with long sequences >​

< Inputs and outputs >​

< In PyTorch >​

< Where it is used >​

Comparison​

< LSTM vs GRU >​

Related ideas​

Reference​