📝 Gradient Checking

Description

< What is it? >

Gradient checking is a debugging technique for verifying a hand-written backpropagation implementation. It compares the gradient from backpropagation with a numerical approximation computed from repeated forward passes.

For parameter vector $\theta$, approximate the $i$th gradient with the central-difference formula:

$$g_{\text{approx},i} = \frac{J(\theta+\epsilon e_i)-J(\theta-\epsilon e_i)}{2\epsilon}$$

Here, $e_i$ changes only parameter $i$, $\epsilon$ is a small number, and $J$ is the loss. Compare the approximation with the backpropagation gradient $g$ using relative difference:

$$\text{difference} = \frac{\lVert g-g_{\text{approx}}\rVert_2} {\lVert g\rVert_2+\lVert g_{\text{approx}}\rVert_2}$$

Key points

< How to use it >

1. Compute gradients with backpropagation.
2. Flatten the model parameters and gradients into vectors.
3. Perturb one parameter at a time by $+\epsilon$ and $-\epsilon$, and run the forward pass for each.
4. Compute $g_{\text{approx}}$ and compare it with the backpropagation gradient.

A very small relative difference gives evidence that the implementation is correct. For a small float64 test, approximately $10^{-7}$ or lower is often reassuring, but the appropriate threshold depends on the loss scale, parameter scale, and $\epsilon$.

< When to use it >

• Use it while implementing or changing backpropagation, custom layers, or custom losses.
• Use a small model and a small batch: checking every parameter needs two extra forward passes per parameter, so it is too slow for ordinary training.
• Turn it off after debugging. Training should use backpropagation, not numerical gradients.

< Dropout and non-smooth operations >

Gradient checking assumes that $J(\theta+\epsilon e_i)$ and $J(\theta-\epsilon e_i)$ evaluate the same deterministic function.

• Disable dropout while checking, or otherwise ensure the same mask is used for both evaluations.
• Be careful near ReLU's zero boundary and other non-smooth points, where a numerical derivative can disagree even when backpropagation is correct.

Related ideas

• Backpropagation calculates the efficient analytical gradient used in training.
• Gradient Descent uses those gradients to update parameters.
• Vector norms explains the norm used in the relative-difference check.

Implementation

< Notebook >

• gradient checking (github)

Reference

• Introduction To Adam Optimizer (geeksforgeeks)
• Explaining Adam & Momentum for Gradient Descent Optimization (Paige Caskey)