📝 Overfitting

Description

< What is it? >

Overfitting is what happens when a model learns the training set rather than the pattern behind it. Training loss keeps falling while validation loss flattens and then climbs: the extra capacity is being spent memorizing noise, which does not transfer to unseen data. In bias–variance terms it is the high-variance regime — see Bias & Variance.

Model complexity vs Loss
Fitting: Under, Good, Over

The response has three parts: notice it (monitoring), limit capacity (regularization), and stop in time (early stopping).

Key points

< Monitoring overfitting >

Signal	How to watch it	What it tells you
Learning curves	Plot training and validation loss per epoch — `tensorboardX` or `torch.utils.tensorboard` give live plots	The gap between the two curves is the overfitting; the epoch where validation turns is where to stop
Validation metrics	Score a held-out validation set every epoch — `torchmetrics` computes accuracy, precision, recall and friends incrementally	Loss can improve while the metric you care about does not; track both
Unseen data	A holdout test set, or cross-validation when data is scarce	The only honest estimate of generalization — see Evaluation Metrics
Capacity vs. data	Compare parameter count to training-set size, and watch the weight norm $\lVert\theta\rVert$ grow during training	A model with far more parameters than examples has the room to memorize; a weight norm that keeps climbing after validation loss bottoms out is that memorization happening

The test set stays sealed while you do this. Tuning against it turns it into a second validation set and the generalization estimate stops being an estimate.

< Regularization techniques >

Each one trades a little bias for a large reduction in variance — Regularization covers the mechanisms in full.

Technique	What it does	Notes
Dropout	Randomly zeroes each unit with probability $p$ during training	`p = 0.1` in transformer blocks, `0.5` in wide fully connected layers
Weight decay	Penalizes the magnitude of the weights, keeping the fitted function smooth	`1e-2` with AdamW, `1e-4` with SGD; see why it is not plain L2 under Adam
Data augmentation	Rotation, crop, zoom, noise — encodes invariances as extra training data	Lowers variance without adding bias, unlike a penalty term
Batch normalization	Normalizes each layer's activations across the mini-batch	Its minibatch noise regularizes mildly as a side effect; not a substitute for the above
Simpler architecture	Fewer layers or narrower ones	The most direct capacity cut, and the first thing to try when the dataset is small

< Hyperparameter optimization >

Learning rate schedules. Decaying the rate over time, or on a validation plateau, lets training settle into a minimum instead of bouncing around it — the single most impactful knob, see Gradient Descent.
Optimization algorithm. Adam, RMSprop, and SGD with momentum reach different minima; AdamW is the usual default when weight decay is in play.
Regularization strength. The L2 coefficient $\lambda$ and the dropout probability $p$ are themselves hyperparameters — tune them on validation data, never on test data. Both directions have a signature: symptoms of getting λ wrong.

< Early stopping >

Stop training at the validation minimum. Fewer steps means less of the model's capacity is actually used, which makes early stopping a regularizer in its own right — provably close to L2 for linear models.

Monitor validation loss. Halt when it has not improved for a set number of epochs.
Use patience. Validation loss is noisy; requiring several bad epochs in a row (plus a min_delta threshold) avoids stopping on a single unlucky fluctuation.
Save the best model, not the last one. The final epoch is by definition past the minimum. Checkpoint whenever validation improves, and deploy that checkpoint.

< Code: early stopping in PyTorch >

import torch
import torch.nn as nn
import torch.optim as optim

train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=64, shuffle=True)
val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=64, shuffle=False)

model = YourModel()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.001)

# Early stopping criteria
patience = 3        # epochs without improvement before halting
min_delta = 0.001   # improvement smaller than this does not count
epochs = 100

best_val_loss = float("inf")
counter = 0

for epoch in range(epochs):
    train_loss = train(model, train_loader, criterion, optimizer)
    val_loss = validate(model, val_loader, criterion)

    if val_loss < best_val_loss - min_delta:
        best_val_loss = val_loss
        counter = 0
        # Checkpoint here: the last epoch is past the minimum, this one is at it.
        torch.save(model.state_dict(), "best_model.pth")
    else:
        counter += 1
        if counter >= patience:
            print(f"Early stopping at epoch {epoch}")
            break

# Load the best checkpoint for evaluation or deployment
best_model = YourModel()
best_model.load_state_dict(torch.load("best_model.pth", weights_only=True))
best_model.eval()

model.eval() matters at the end: it switches dropout off and makes batch norm use its running statistics, so inference is deterministic.

Bias & Variance places overfitting on the capacity curve and gives the diagnostic table.
Regularization is the toolbox of fixes, with the mechanism behind each.
Training, Validation & Test Sets is what makes the measurement trustworthy.
Vanishing & Exploding Gradients is the opposite failure — a model that cannot fit the training set at all.

Description​

< What is it? >​

Key points​

< Monitoring overfitting >​

< Regularization techniques >​

< Hyperparameter optimization >​

< Early stopping >​

< Code: early stopping in PyTorch >​

< Related ideas >​

Reference​