📝 Dropout Layer

Description

< What is dropout? >

Dropout randomly sets some hidden activations to zero during training. It is a regularization technique for neural networks—not an optimizer or an initialization method.

< Drop probability and keep_prob >

If the dropout probability is $p$, each activation is dropped with probability $p$ and kept with probability $1-p$:

$$\text{keep probability} = 1 - p \qquad \text{dropout rate} = 1 - \text{keep\_prob}$$

For example, p = 0.2 drops 20% of activations; equivalently, keep_prob = 0.8 keeps 80%. Tutorials often use keep_prob, while PyTorch's Dropout(p=0.2) and TensorFlow's rate=0.2 use the probability of dropping a unit.

Key points

< Training, inference, backpropagation >

For an input vector $x$, dropout samples a binary mask $m$ for every forward pass. Each element $m_i$ is 1 with probability $1-p$ and 0 with probability $p$:

$$m_i \sim \operatorname{Bernoulli}(1-p)$$

With inverted dropout, the forward and backward rules are:

$$\begin{aligned} \text{training:}\qquad & y = \frac{x \odot m}{1-p} \\ \text{inference:}\qquad & y = x \\ \text{backpropagation:}\qquad & \frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y} \odot \frac{m}{1-p} \end{aligned}$$

< What the backward pass receives and returns >

Object	What it is	Where it comes from
$\frac{\partial \mathcal{L}}{\partial y}$	How the loss changes with dropout's output	Received from the next layer (grad_output)
$\frac{\partial y}{\partial x} = \frac{m}{1-p}$	Dropout's local Jacobian	Computed from the cached mask
$\frac{\partial \mathcal{L}}{\partial x}$	How the loss changes with dropout's input	Computed here and sent to the previous layer (grad_input)

The chain rule wires them together:

$$\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y} \odot \frac{\partial y}{\partial x}.$$

Every layer's backward pass is a small function that receives the gradient with respect to its output and returns the gradient with respect to its input:

forward:   h ──[linear]──▶ x ──[dropout]──▶ y ──[next layer]──▶ … ──▶ L
backward:  ∂L/∂h ◀──[linear]── ∂L/∂x ◀──[dropout]── ∂L/∂y ◀── … ◀── 1

The same mask is reused for its matching backward pass, so an activation dropped in the forward pass receives no gradient for that pass. Since $\mathbb{E}[m_i] = 1-p$, scaling by $1/(1-p)$ makes $\mathbb{E}[y_i] = x_i$; evaluation can therefore use every unit without additional scaling.

< A small example >

Suppose a layer produces:

$$x = [1, 2, 3, 4]$$

With $p = 0.5$, one possible mask is $m = [1, 0, 1, 0]$. In that training pass, inverted dropout produces:

$$\frac{x \odot m}{1-p} = \frac{[1, 2, 3, 4] \odot [1, 0, 1, 0]}{0.5} = [2, 0, 6, 0]$$

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(512, 512),
    nn.ReLU(),
    nn.Dropout(p=0.2),  # p is the probability of dropping a unit
)

model.train()  # dropout is active
model.eval()   # dropout is disabled

< Why it regularizes >

• It prevents co-adaptation: a unit must learn features that remain useful with many different subsets of other units.
• Each random mask trains a different thinned subnetwork, acting like a lightweight ensemble whose parameters are shared.
• The temporarily reduced capacity makes memorizing the training set harder and can reduce overfitting.
• Dropout does not delete neurons or parameters; it only masks their outputs for an individual training pass.

< Practical guidance >

• Add dropout to hidden representations when training performance is much better than validation performance.
• Start modestly: p = 0.1 is common in transformer blocks; p = 0.5 is more typical for wide fully connected layers.
• Do not use dropout to fix underfitting, and do not forget model.eval() for evaluation or inference.

Favorites

🤿

Magic

Reference

• Dropout: A Simple Way to Prevent Neural Networks from Overfitting — Srivastava et al. (2014)
• PyTorch Dropout documentation