Skip to main content

πŸ“ Regularization

Description​

< What is it? >​

  • [The idea]

    • Regularization is any deliberate constraint on a model that trades a little bias for a large drop in variance β€” see Bias & Variance. It is the standard answer to overfitting: the model class stays flexible, but the training objective is changed so that flexibility has a price.

    • Unregularized training minimizes the fit alone. Regularized training minimizes the fit plus a penalty on the parameters:

      J(ΞΈ)  =  L(ΞΈ)  +  Ξ» Β· R(ΞΈ)
               β”‚         β”‚   β”‚
          data loss      β”‚   penalty on the parameters
                    strength knob (Ξ» β‰₯ 0)

    • Ξ» = 0 recovers the unregularized model (low bias, high variance). Ξ» β†’ ∞ crushes every weight toward zero (high bias, low variance). The useful value sits in between and is found by cross-validation, never by training error.

    • Weight decay
      Weight decay is a regularization technique that penalizes large weights in a neural network to prevent overfitting. It adds an extra term to the loss functionβ€”typically the squared magnitude (L2 norm) of the weights. During optimization, this effectively shrinks the weights by a small factor at each update step (multiplying them by (1 - learning_rate * decay_factor) before applying the gradient). In PyTorch, you set it directly in the optimizer, e.g.:

      optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=1e-4)
  • [Why it works]
    • A model overfits by using large, finely balanced weights to thread every training point, including the noise. Penalizing weight magnitude removes that option: the model can only spend "weight budget" on structure that pays for itself across many examples.
    • Equivalently, the penalty is a prior. L2 corresponds to a Gaussian prior on the weights, L1 to a Laplace prior; regularized training is MAP estimation rather than maximum likelihood.
    • Why regularization reduces overfitting (Andrew Ng, deeplearning.ai)
      A large Ξ» drives W toward 0, which zeroes out hidden units and moves the fit from high variance back toward high bias

      A large Ξ» pushes W[l] β‰ˆ 0, which effectively removes hidden units β€” the network collapses toward a much smaller one, sliding the fit from "high variance" back through "just right".

    • Why does drop-out work (Andrew Ng, deeplearning.ai) dropout

< The two classic penalties >​

L2 (Ridge / weight decay)      R(ΞΈ) = Ξ£ ΞΈβ±ΌΒ²        β†’  shrinks all weights smoothly toward 0
L1 (Lasso)                     R(ΞΈ) = Ξ£ |ΞΈβ±Ό|       β†’  drives many weights exactly to 0
Elastic Net                    R(ΞΈ) = Ξ± Ξ£|ΞΈβ±Ό| + (1βˆ’Ξ±) Ξ£ ΞΈβ±ΌΒ²
  • L2 has a smooth gradient (2ΞΈ), so each update multiplies the weights by a factor slightly below 1 β€” hence the name weight decay. Correlated features share the weight between them rather than one winning outright.
  • L1 has a constant-magnitude gradient (sign(ΞΈ)), so a weight is pushed toward zero at the same rate no matter how small it already is, and it lands exactly on zero. The result is feature selection: the fitted model ignores whole inputs.
  • Elastic Net keeps L1's sparsity while L2 stabilizes the choice among correlated features, which L1 alone makes arbitrarily.

< Why L1 produces sparsity >​

  • Read the penalty as a constraint region: minimize the loss subject to Ξ£|ΞΈβ±Ό| ≀ t (L1) or Σθⱼ² ≀ t (L2). The solution is where the loss contours first touch that region.
  • The L1 region is a diamond with corners on the axes; the L2 region is a circle with no corners. Contours coming in from an arbitrary direction usually touch a diamond at a corner β€” and a corner means some coordinate is exactly zero. A circle is touched at a generic point, where every coordinate is small but nonzero.

Key points​

< Choosing a penalty >​

PenaltyEffect on weightsUse when
L2 / ridgeAll shrunk, none removedDefault. Many weakly-informative features; correlated inputs
L1 / lassoMany set to exactly 0You want a sparse, interpretable model or automatic feature selection
Elastic NetSparse, but stable under correlationWide data (p ≫ n) with correlated feature groups
Noneβ€”The model is already underfitting, or data volume vastly exceeds capacity

< Practical rules that are easy to get wrong >​

  1. [Standardize first] the penalty is scale-sensitive β€” a feature measured in millimetres gets a larger weight, and so a larger penalty, than the same feature in metres. Standardize (or normalize) before fitting, or the penalty silently encodes your unit choices.
  2. [Do not penalize the intercept] the bias term ΞΈβ‚€ only shifts the output; shrinking it biases predictions toward zero for no benefit. Standard libraries exclude it β€” custom loops often forget.
  3. [Tune Ξ» on validation data] a log-spaced sweep (1e-4 … 1e2) under k-fold CV. Selecting Ξ» by test error turns the test set into a training signal.
  4. [Ξ» scales with the data] a penalty tuned on 10k rows is usually too strong on 1M rows, because the data loss term grows while the penalty does not.

< Regularization beyond the penalty term >​

TechniqueWhere it appliesMechanism
Early stoppingAny iterative trainingStop at the validation-error minimum; less training time = less capacity actually used.
Provably similar to L2 for linear models.
For many loss landscapes (especially in the linear or neural tangent kernel regimes), early stopping is mathematically equivalent to L2 regularization (Tikhonov regularization). In L2 regularization, you explicitly penalize large weights by adding βˆ₯ΞΈβˆ₯Β² to the loss function, forcing weights to stay small. In early stopping, you take fewer gradient descent steps. Because the gradient is multiplied by the learning rate at each step, the total movement of the weights is bounded. This implicitly restricts the norm (size) of the weight vector. Smaller weights produce smoother, less sensitive decision boundaries, which directly combats overfitting.
DropoutNeural networksRandomly zero a fraction of units per step, so no unit can depend on a specific co-adapted partner. Acts like averaging an ensemble of thinned networks
Data augmentationVision, audio, textEncodes invariances (a rotated cat is a cat) as extra data β€” lowers variance without adding bias (rotation, zoom in/out, crop, etc)
More dataEverythingThe one "regularizer" with no bias cost at all
Ensembling / baggingTrees, any high-variance modelAveraging independent errors β€” see Random Forest
Label smoothingClassificationTargets of 0.9/0.1 instead of 1/0 stop the network from driving logits to extremes
Pruning / max depthDecision treesStructural capacity limit; XGBoost adds explicit Ξ», Ξ± and Ξ³ terms to the split objective

< In deep learning >​

  • Weight decay β‰  L2 under Adam. Adding Ξ»β€–ΞΈβ€–Β² to the loss makes the penalty pass through Adam's per-parameter scaling, so heavily-updated weights get decayed less. AdamW applies the decay directly to the weights instead, outside the adaptive step β€” which is why it is the default optimizer for transformers.
  • Typical starting points: weight decay 1e-2 (AdamW, transformers) or 1e-4 (SGD, vision), dropout 0.1 in transformer blocks and 0.5 in wide fully-connected layers.
  • Batch norm / layer normalization is not regularization by design, but batch norm's dependence on the minibatch injects noise that has a mild regularizing side effect β€” one reason it and dropout are often not both needed.
  • At very large scale the picture shifts: with enough data relative to parameters the model no longer has spare capacity to memorize, and heavy explicit regularization mostly costs accuracy. Scale is itself a variance reducer.

< Symptoms of getting Ξ» wrong >​

ObservationLikely causeMove
Train error low, validation much higherToo little regularizationRaise Ξ», add dropout, get more data
Train and validation error both high, close togetherToo much regularizationLower Ξ» β€” you have induced bias
Validation error dips then climbs during trainingOverfitting in progressEarly stopping at the dip
Weights all near zero, predictions near the meanΞ» far too largeLower by an order of magnitude

Reference​