📝 Regularization

Description

< What is it? >

• [The idea]

  • Regularization is a strategy for reducing a model's effective capacity or sensitivity, usually trading some bias for lower variance — see Bias & Variance. It is a common response to overfitting.

  • Many regularizers add a penalty to the training objective. Others, such as early stopping, data augmentation, and dropout, regularize in different ways:

    J(θ)  =  L(θ)  +  λ · R(θ)
             │         │   │
        data loss      │   penalty on the parameters
                  strength knob (λ ≥ 0)

  • Linear regression regularization: For $\hat{y}^{(i)} = w^\top x^{(i)} + b$, the common objectives are:

    • L1 regularization for linear regression.
    $$\begin{aligned} J_{\text{lasso}}(w, b) &= \underbrace{ \frac{1}{2m}\sum_{i=1}^{m}\left(\hat{y}^{(i)}-y^{(i)}\right)^2 }_{\text{squared-error cost}} + \underbrace{ \frac{\lambda}{m}\lVert w\rVert_1 }_{\text{L1 regularization cost}} \end{aligned}$$

    L1-regularized linear regression is lasso regression.

    • L2 regularization for linear regression.
    $$\begin{aligned} J_{\text{ridge}}(w, b) &= \underbrace{ \frac{1}{2m}\sum_{i=1}^{m}\left(\hat{y}^{(i)}-y^{(i)}\right)^2 }_{\text{squared-error cost}} + \underbrace{ \frac{\lambda}{2m}\lVert w\rVert_2^2 }_{\text{L2 regularization cost}} \end{aligned}$$

    L2-regularized linear regression is ridge regression. Libraries may place constant factors such as $1/2$ or $1/m$ differently, so tune $\lambda$ for the implementation rather than comparing its numeric value across libraries.

  • $\lambda = 0$ recovers the unregularized objective. As $\lambda$ becomes very large, penalized weights tend toward zero and the model can underfit. Select a useful value with cross-validation, not training error.

  • Logistic regression regularization: for binary logistic regression, $p^{(i)} = \sigma(w^\top x^{(i)} + b)$ is the predicted probability that $y^{(i)}=1$.

    • L1 regularization for logistic regression. Keep the same binary cross-entropy term, but penalize the absolute values of the weights instead:

      $$J_{\text{L1}}(w, b) = \underbrace{ -\frac{1}{m}\sum_{i=1}^{m} \left[ y^{(i)}\log p^{(i)} + \left(1-y^{(i)}\right)\log\left(1-p^{(i)}\right) \right] }_{\text{cross-entropy cost}} + \underbrace{ \frac{\lambda}{m}\lVert w\rVert_1 }_{\text{L1 regularization cost}}$$

      This is L1-regularized logistic regression. Its optimized solution can set some weights exactly to zero, so it is useful when sparse feature selection matters.

    • L2 regularization for logistic regression. The L2-regularized objective is:

      $$J(w, b) = \underbrace{ -\frac{1}{m}\sum_{i=1}^{m} \left[ y^{(i)}\log p^{(i)} + \left(1-y^{(i)}\right)\log\left(1-p^{(i)}\right) \right] }_{\text{cross-entropy cost}} + \underbrace{ \frac{\lambda}{2m}\lVert w\rVert_2^2 }_{\text{L2 regularization cost}}$$

      The first term is binary cross-entropy; the second is the L2 penalty on the weight vector. By common convention, the intercept $b$ is not penalized, although library behavior can differ. A neural network uses the same idea, but applies a norm-based penalty to the weight matrices $W^{[l]}$ across its layers.

  • Weight decay
    Weight decay helps reduce overfitting by discouraging large weights. With plain SGD, an L2 penalty and weight decay are equivalent after their coefficients are matched. With adaptive optimizers such as Adam, use a decoupled optimizer such as AdamW when you want an explicit shrinkage step.

    The coefficient $\lambda$ (or weight_decay) is a hyperparameter tuned on a development or validation set. L2 assumes that, all else equal, a model with smaller weights is simpler. Penalizing squared weights makes large weights costly, typically making the prediction function or decision boundary smoother: its output changes less sharply as the input changes. If $\lambda$ is too large, it can oversmooth the model and cause underfitting (high bias).

    In PyTorch, for example:

    optimizer = torch.optim.AdamW(
        model.parameters(),
        lr=1e-3,
        weight_decay=1e-4,
    )

• [Why it works]
  • A model can overfit by using finely balanced, high-magnitude weights to fit individual training points, including noise. Penalizing weight magnitude makes that option more costly, so the model must spend its "weight budget" on structure that pays for itself across many examples.
  • When the data loss is a negative log-likelihood, an L2 penalty corresponds to a Gaussian prior on the weights and an L1 penalty to a Laplace prior. Under this interpretation, regularized training is MAP estimation rather than maximum likelihood.
  • Why regularization reduces overfitting (Andrew Ng, deeplearning.ai)
    Shrinking weights ≈ a simpler network

    

    A large $\lambda$ makes $W^{[l]}$ small, so some units can contribute very little and the network behaves like a lower-capacity model. L2 regularization does not normally set weights exactly to zero.

    Small weights ≈ near-linear layers

    

    When inputs and biases are appropriately scaled, smaller $W^{[l]}$ can keep $z^{[l]}$ nearer tanh's central, responsive region instead of its saturated tails, where gradients approach zero. This is a helpful heuristic for sigmoid/tanh networks, not a guarantee; ReLU networks have a different saturation pattern.

  • Why does drop-out work (Andrew Ng, deeplearning.ai)

< The two classic penalties >

L2 (Ridge / L2 penalty)          R(θ) = Σ θⱼ²        →  shrinks all weights smoothly toward 0
L1 (Lasso)                     R(θ) = Σ |θⱼ|       →  drives many weights exactly to 0
Elastic Net                    R(θ) = α Σ|θⱼ| + (1−α) Σ θⱼ²

• L2 has a smooth gradient (2θ). With plain SGD, its update includes a shrinkage factor slightly below 1 — hence the name weight decay. Adaptive optimizers need the L2-versus-decoupled-weight-decay distinction described above.
• L1 has subgradient sign(θ) away from zero and a kink at zero. Its optimized solution often has exact zeros; proximal or coordinate-descent methods make that sparsity especially explicit.
• Elastic Net keeps L1's sparsity while L2 stabilizes the choice among correlated features, for which L1 alone can be unstable.

< Why L1 produces sparsity >

• Read the penalty as a constraint region: minimize the loss subject to Σ|θⱼ| ≤ t (L1) or Σθⱼ² ≤ t (L2). The solution is where the loss contours first touch that region.
• The L1 region is a diamond with corners on the axes; the L2 region is a circle with no corners. Contours coming in from an arbitrary direction usually touch a diamond at a corner — and a corner means some coordinate is exactly zero. A circle is touched at a generic point, where every coordinate is small but nonzero.

Key points

< Choosing a penalty >

Penalty	Effect on weights	Use when
L2 / ridge	All shrunk, none removed	Default. Many weakly-informative features; correlated inputs
L1 / lasso	Many set to exactly 0	You want a sparse, interpretable model or automatic feature selection
Elastic Net	Sparse, but stable under correlation	Wide data (p ≫ n) with correlated feature groups
None	—	The model is already underfitting, or validation shows no useful benefit from an explicit penalty

< Practical rules that are easy to get wrong >

1. [Standardize first] the penalty is scale-sensitive — the same relationship expressed in millimetres needs a coefficient 1,000 times smaller than when expressed in metres, changing its L1/L2 penalty. Standardize (or normalize) before fitting, or the penalty silently encodes your unit choices.
2. [Usually do not penalize the intercept] the bias term θ₀ shifts the output baseline, so it is commonly excluded from the penalty. Check the library or solver, because conventions differ.
3. [Tune λ on validation data] a log-spaced sweep (1e-4 … 1e2) under k-fold CV. Selecting λ by test error turns the test set into a training signal.
4. [Keep the objective convention straight] in the averaged objectives above, both the data loss and penalty are scaled by 1/m, so their relative strength does not automatically change just because rows are added. More representative data can still change the best λ, so retune it on validation data.

< Regularization beyond the penalty term >

Technique	Where it applies	Mechanism
Early stopping	Any iterative training	Stop at the validation-error minimum. It limits fitting; for linear models with quadratic loss and simple gradient descent, it is closely related to L2, but generally distinct
Dropout	Neural networks	Randomly zero a fraction of units per step, reducing co-adaptation; it can be interpreted as approximate averaging over thinned networks
Data augmentation	Vision, audio, text	Adds label-preserving transformations to encode useful invariances. Wrong augmentations can introduce bias
More data	Everything	More independent, representative data often reduces variance. It is not an explicit regularizer and does not guarantee an improvement
Ensembling / bagging	Trees, any high-variance model	Averages imperfectly correlated errors — see Random Forest
Label smoothing	Classification	Discourages extreme logits; for multiclass tasks it gives a small amount of target probability to other classes
Pruning / max depth	Decision trees	Structural capacity limit; XGBoost adds explicit λ, α, and γ terms to the split objective

< In deep learning >

• Weight decay and L2 differ under Adam. A coupled L2 penalty gradient passes through Adam's adaptive per-parameter scaling, whereas AdamW applies a separate decay step. Their effective strength can therefore differ by parameter and optimization history. AdamW is common for transformers, not a universal default.
• Typical values are rough starting points, not defaults: weight decay 1e-2 (AdamW, transformers) or 1e-4 (SGD, vision), dropout 0.1 in transformer blocks, and 0.5 in wide fully connected layers. Tune them on validation data.
• Batch norm / layer normalization is not regularization by design. Batch norm's minibatch noise can have a mild regularizing side effect, but it and dropout can be combined when validation results support it.
• More representative data often reduces variance, but even overparameterized models can memorize. Explicit regularization can help or hurt, so choose it with validation data.

< Symptoms of getting λ wrong >

These are clues rather than diagnoses: data quality, distribution shift, and optimization problems can produce similar patterns.

Observation	Likely cause	Move
Train error low, validation much higher	Too little regularization	Raise λ, add dropout, get more data
Train and validation error both high, close together	Too much regularization	Lower λ — you have induced bias
Validation error dips then climbs during training	Overfitting in progress	Early stopping at the dip
Weights all near zero, predictions near the mean	λ far too large	Lower by an order of magnitude

Implementation

• see here

Reference

• Regularization in Machine Learning (GeeksforGeeks)
• Regularization (math) (Wikipedia)
• Deep Learning, ch. 7 — Regularization (Goodfellow, Bengio, Courville)
• Linear models and regularization in scikit-learn (scikit-learn docs)
• The Elements of Statistical Learning, ch. 3 — Shrinkage Methods (Hastie, Tibshirani, Friedman)
• Regression Shrinkage and Selection via the Lasso (Tibshirani, 1996)
• Dropout: A Simple Way to Prevent Neural Networks from Overfitting (Srivastava et al., 2014)
• Decoupled Weight Decay Regularization — the AdamW paper (Loshchilov & Hutter, 2019)
• PyTorch Adam optimizer documentation (PyTorch)