Skip to main content

📝 Backpropagation

Description​

< What is it? >​

Backpropagation is the algorithm that efficiently calculates how the loss changes with every trainable parameter in a neural network. It runs from the output layer back to the input, repeatedly applying the chain rule.

It has two passes:

forward:   input → layers → prediction → loss
backward:  loss  → gradients for each layer and parameter

Backpropagation computes gradients; an optimizer such as SGD or Adam uses them to update the parameters.

Key points​

< The computation graph >​

The flat sketch above is a simplification. What the forward pass actually leaves behind is a computation graph: a directed acyclic graph whose nodes are operations and whose edges carry the values flowing between them. Every elementary step — a matrix multiply, an addition, a ReLU — becomes a node that also remembers how to differentiate itself.

   x ──▶[ matmul ]──▶ u ──▶[ + b ]──▶ z ──▶[ ReLU ]──▶ a ──▶[ loss ]──▶ 𝓛
   W ──▶    │                 ▲
            └── forward: values move right ──┘
   ◀────────  backward: gradients move left, one local derivative per node

The backward pass walks that graph in reverse topological order — no node is visited until every node that consumes its output has been. At each one it does the same small thing: take the gradient arriving from downstream, multiply by the node's own local derivative, and pass the result upstream. That repeated multiply-and-pass is the chain rule; backpropagation is just the bookkeeping that applies it in an efficient order.

Two consequences follow from the structure:

  • Fan-out sums. A value used by two downstream nodes receives a gradient from each, and they add. This is why gradients accumulate rather than overwrite — and why the training loop must call zero_grad() between steps.
  • One backward pass covers every parameter. Walking the graph backward computes ∂L/∂θ\partial\mathcal{L}/\partial\theta for all parameters at once, at roughly the cost of one forward pass. Differentiating forward instead would need a separate pass per parameter — for a million-parameter model, a million passes. That asymmetry is the whole reason reverse-mode is what trains neural networks.

The graph is built from the actual operations that ran, so anything expressible in code is differentiable — including branches and loops whose shape changes per batch. In PyTorch this graph is autograd, each tensor's grad_fn is its node, and loss.backward() is the traversal — see PyTorch for requires_grad, no_grad(), and the dynamic-vs-static distinction.

< One layer >​

For a layer z=Wx+bz = Wx + b followed by an activation a=f(z)a = f(z), suppose the next layer supplies ga=∂L∂ag_a = \frac{\partial \mathcal{L}}{\partial a}. Backpropagation calculates:

gz=ga⊙f′(z)∂L∂W=gzx⊤∂L∂b=gz∂L∂x=W⊤gz\begin{aligned} g_z &= g_a \odot f'(z) \\ \frac{\partial \mathcal{L}}{\partial W} &= g_zx^\top \\ \frac{\partial \mathcal{L}}{\partial b} &= g_z \\ \frac{\partial \mathcal{L}}{\partial x} &= W^\top g_z \end{aligned}

The last quantity, ∂L∂x\frac{\partial \mathcal{L}}{\partial x}, becomes the gradient supplied to the previous layer.

< What is saved for the backward pass? >​

Each node keeps whatever its own local derivative needs, and nothing else: for example, an activation layer stores its input or output, and a dropout layer stores its random mask. Reusing the correct cached values makes the backward calculation match the forward pass.

< In PyTorch >​

loss = loss_fn(prediction, target)
loss.backward()   # computes parameter gradients
optimizer.step()  # uses them to update parameters
optimizer.zero_grad()

< Related ideas >​

Crash course​

  • computation graph

Reference​