📝 Vanishing & Exploding Gradients Problems

Description

< The chain rule >

The chain rule differentiates a function nested inside another. If $y$ depends on $u$, and $u$ depends on $x$:

$$\frac{dy}{dx} = \frac{dy}{du}\frac{du}{dx}$$

For a longer chain, consider:

$$x \longrightarrow h_1 = 3x + 1 \longrightarrow h_2 = h_1^2 \longrightarrow h_3 = \tanh(h_2) \longrightarrow y = 2h_3 = 2\tanh\left((3x + 1)^2\right)$$

The derivative travels back through every step:

$$\begin{aligned} \frac{dy}{dx} &= \underbrace{2}_{\frac{\partial y}{\partial h_3}} \cdot \underbrace{\left(1-\tanh^2(h_2)\right)}_{\frac{\partial h_3}{\partial h_2}} \cdot \underbrace{2h_1}_{\frac{\partial h_2}{\partial h_1}} \cdot \underbrace{3}_{\frac{\partial h_1}{\partial x}} \\ &= \frac{\partial y}{\partial h_3} \cdot \frac{\partial h_3}{\partial h_2} \cdot \frac{\partial h_2}{\partial h_1} \cdot \frac{\partial h_1}{\partial x} \end{aligned}$$

Differentiate the outside while keeping the inside expression

A longer nested function creates one derivative factor per layer

Neural-network layers form the same kind of nested chain; backpropagation repeatedly applies this rule to determine how every parameter affects the loss.

The vanishing-gradient problem is a consequence of repeatedly applying the chain rule across many layers or time steps: every application introduces another derivative factor, and those factors are multiplied together. If they are usually smaller than 1, the gradient shrinks toward zero; if they are usually larger than 1, it can explode.

< What are they? >

Backpropagation sends a loss signal from the output back through every layer. For an earlier representation $h_\ell$, that signal contains a chain of Jacobians:

$$\frac{\partial \mathcal{L}}{\partial h_\ell} = \frac{\partial \mathcal{L}}{\partial h_L} \prod_{k=\ell+1}^{L} \frac{\partial h_k}{\partial h_{k-1}}$$

In a deep network—or a recurrent network unrolled across many time steps—repeated multiplication is the problem:

Problem	What happens	Result
Vanishing gradients	Typical factors are smaller than 1, so the signal shrinks toward zero.	Early layers barely update; long-range dependencies are hard to learn.
Exploding gradients	Typical factors are larger than 1, so the signal grows without bound.	Updates become unstable; loss or gradients may become inf / NaN.

This is about the backward signal, not merely a high or low loss. A model can have a finite loss while some layers receive almost no useful gradient.

Key points

< When it happens >

Deep networks and recurrent networks repeatedly multiply local derivatives. The same basic mechanism appears in several practical settings.

< Vanishing gradients >

Scenario	Why gradients vanish	Typical fix
Deep sigmoid / tanh networks	Sigmoid derivatives are at most $0.25$; repeated factors such as $0.25^{10}\approx9.5\times10^{-7}$ erase early-layer signal. Tanh also saturates away from zero.	ReLU-family activations and initialization matched to the activation
Vanilla RNNs on long text or time series	The recurrent Jacobian is multiplied once per time step. Typical factors below $1$ erase distant context.	LSTM or GRU gates
Very deep networks without residual paths	Even non-saturating activations can gradually shrink a signal across hundreds of layers.	Residual connections, normalization, and suitable initialization
Dead or saturated units	A ReLU held below zero has a local derivative of $0$; saturated sigmoid / tanh units have near-zero derivatives.	Leaky ReLU or ELU, plus He initialization
Historical deep autoencoders / belief networks	Bottom layers often received too little end-to-end signal from random initialization.	Modern residual designs, normalization, and initialization replaced much layer-wise pretraining

< Exploding gradients >

Scenario	Why gradients explode	Typical fix
Very deep networks with poor initialization or no normalization	Repeated Jacobians with large singular values amplify the backward signal layer by layer.	He / Kaiming initialization for ReLU, normalization, and residual paths
Vanilla RNNs on long sequences	The same recurrent matrix is reused at every time step; factors above $1$ grow exponentially across time.	LSTM or GRU, global-norm clipping, and sometimes truncated BPTT
Early GAN training	An imbalanced generator and discriminator can create sharp, unstable generator updates.	Balanced learning rates, gradient penalties or Wasserstein-style objectives, and clipping when needed
Regression with extreme outliers	MSE gradients grow with the prediction error, so one extreme target can dominate an update.	Target transforms, outlier handling, or Huber loss
Policy-gradient reinforcement learning	High-variance or unusually large returns can create large policy-gradient updates.	Global-norm clipping, advantage normalization, or PPO's clipped objective

Long, deep Transformer training can also produce early gradient spikes, so global-norm clipping is common in training recipes.

< How to diagnose it >

Exploding gradients are usually easier to spot: norms spike, the loss becomes unstable, or values become inf / NaN. Vanishing gradients are subtler: they remain finite but can be too small to train early layers. Slow learning, a flat loss, or poor long-range learning can also have other causes.

• Log raw gradient norms layer by layer over time (for example, $\lVert \nabla W \rVert$), not only parameter updates: momentum or Adam can make updates look nonzero even when the current gradient is tiny.
  • Early-layer norms consistently far below later-layer norms suggest vanishing.
  • Sudden huge norms or non-finite values suggest exploding.
• Check activation statistics. Saturated sigmoid or tanh units have derivatives close to zero and can starve earlier layers of gradient.
• Watch the loss curve, but do not rely on it alone: a flat loss can also be caused by a bad learning rate, a disconnected computation graph, or insufficient model capacity.

< How to reduce the problem >

< Vanishing gradients >

• Use an appropriate activation: ReLU, Leaky ReLU, ELU, and GELU avoid the saturated sigmoid / tanh derivative in their responsive regions
• Match the initialization: Xavier / Glorot for tanh-like activations and He / Kaiming for ReLU-like activations
• Add stable paths: residual connections give gradients an identity shortcut; BatchNorm and LayerNorm keep activation scales better conditioned
• Use gated sequence models: LSTM and GRU cells preserve long-range signal more reliably than a vanilla RNN

< Exploding gradients >

• Clip the global norm: cap an unusually large gradient after backward() and before optimizer.step(); this is the first-line defense against spikes
• Control scale from the start: suitable initialization, normalization, and residual paths reduce harmful amplification
• Use a smaller learning rate or warmup: this limits destructive early parameter updates while optimizer statistics stabilize
• Stabilize the task-specific signal: use robust losses or target transforms for outliers; use gates, truncated BPTT, advantage normalization, PPO, or GAN-stabilizing objectives where appropriate

< Why Batch Normalization helps >

Batch normalization helps indirectly: it normalizes a layer's activations, not the gradients themselves. For a mini-batch, it computes:

$$\hat{z}=\frac{z-\mu_{\text{batch}}}{\sqrt{\sigma_{\text{batch}}^2+\epsilon}}, \qquad y=\gamma\hat{z}+\beta$$

• It keeps activation scales more consistent between layers, making one layer less likely to greatly amplify or shrink the signal.
• For sigmoid and tanh, it keeps inputs nearer their responsive range rather than saturated regions, where derivatives are near zero and gradients vanish.
• More stable activation scales make optimization better conditioned, so gradients and parameter updates are less erratic and larger learning rates can often be used safely.
• The learnable $\gamma$ and $\beta$ let the model choose an appropriate scale and offset instead of permanently forcing zero mean and unit variance.

The Batch Normalization page covers the rest — the training/inference split, the PyTorch layers, and batch-size sensitivity.

BatchNorm is not a complete cure: it cannot guarantee that gradients will never vanish or explode. Residual connections, suitable initialization, gated RNNs, and gradient clipping can still be important.

< A practical response >

1. Record layer-wise gradient norms and check for NaN / inf values.
2. For explosions, clip the global norm after backward() and before the optimizer step; then lower the learning rate if needed.
3. For vanishing gradients, revisit initialization, activation functions, and architecture—especially residual paths or gates.
4. With fp16, use loss scaling (or bf16) to avoid numerical underflow. This does not solve mathematical vanishing gradients.

< Related ideas >

• Gradient Descent explains how gradients become parameter updates.
• Data Normalization helps make optimization better conditioned, but it is not a substitute for stable gradient paths through a deep model.

Crash course

Vanishing Gradients

Vanishing AND Exploding Gradient Problem Explained

Vanishing/Exploding Gradients

Examples

💡 The most famous real-world example:

• In the early 2000s, researchers tried to train deep RNNs for speech recognition but hit a wall. The gradients would vanish after ~10-15 time steps, so the model could only use the last few milliseconds of audio, ignoring the broader context. The invention of LSTM (by Hochreiter & Schmidhuber in 1997, but popularized later) was a direct response to this—it used additive cell states to keep the gradient magnitude stable indefinitely, finally allowing RNNs to process entire sentences and audio clips.

Reference

• Understanding the difficulty of training deep feedforward neural networks — Glorot & Bengio (2010)
• On the difficulty of training recurrent neural networks — Pascanu, Mikolov & Bengio (2013)
• Delving Deep into Rectifiers — He et al. (2015)
• Chain rule (Wikipedia)
• The Chain rule (Math Bootcamps)