📝 Bias & Variance

Description

< What is it? >

• [The two error sources]

  Every model's prediction error decomposes into three parts: bias², variance, and irreducible noise. Bias and variance are the two you can trade against each other; the noise is a property of the data and no model removes it.

  • Bias — error from wrong assumptions. The model is too simple to represent the true pattern, so it is consistently wrong in the same direction. Fitting a straight line to a curve is bias.
    • Observation: High training and validation error
  • Variance — error from sensitivity to the particular training sample. The model is flexible enough to chase noise, so it swings wildly when retrained on a different draw of the same data.
    • Observation: Low training error, high validation error
      
    
    Fig 1: Graphical illustration of bias and variance
    

    y = k * x + b
    1. if b is high, high bias, low variance: the model is consistently wrong in the same direction.
    2. if we use y = k * x ^ 2 + b, low bias, high variance. Since output y becomes too large due to x ^ 2, the result spread out too much.

• [Trade-off]
  Increasing model flexibility usually:

  • Reduces bias because the model can represent more complex behavior.
  • Increases variance because the model has more opportunities to learn noise.
    

  Reducing model flexibility usually does the opposite.
  A common decomposition is the squared-error split below.
  The business objective is to choose the point that maximizes out-of-sample value—not necessarily the model with the lowest training loss.

• [Not the bias term]

  • The b in ŷ = θ₀ + θ₁x₁ + … (the intercept, see Linear Regression) is a parameter, unrelated to statistical bias. Same word, different concept — the parameter shifts one model's output, while bias here describes the average error of a whole class of models.

• [Underfitting vs overfitting]

  • Underfitting = high bias. Bad on training data and test data.

  • Overfitting = high variance. Excellent on training data, much worse on test data.

  • The gap between training error and validation error is the practical readout: a large gap means variance, a high floor on both means bias.

    

    Fig 2: The variation of Bias and Variance with the model complexity

• [Examples]

  • 
  • 

< The decomposition >

For squared error at a point, over all possible training sets:

E[(y − ŷ)²]  =  Bias[ŷ]²  +  Var[ŷ]  +  σ²
                  │            │        │
      wrong assumptions   sensitivity   irreducible
                          to the sample    noise

• Increasing model complexity lowers bias and raises variance. Decreasing it does the reverse. The total is minimized somewhere in between — that point is what tuning is searching for.
• The tradeoff is not a law that complexity always hurts: more data lowers variance without raising bias, which is why large models trained on enough data escape the classical picture.

Key points

< Diagnosing which one you have >

Symptom	Training error	Validation error	Diagnosis
Both high, gap small	High	High	High bias — underfitting
Training low, gap large	Low	High	High variance — overfitting
Both low, gap small	Low	Low	Well fit — the target
Training high, validation lower	High	Lower	Data leak or a bug, not a tradeoff

• Plot both curves against training-set size (a learning curve). Curves converging at a high error means bias — more data will not help. A persistent gap means variance — more data will.

< Fixes for high bias >

1. [More capacity] a richer model class — more features, polynomial terms, a deeper network, more trees.
2. [Better features] feature engineering that exposes the real structure the current model cannot express.
3. [Train longer] an undertrained model is underfit for reasons that look identical from the error curves.
   • Use the same training dataset for more iterations (epochs), or a lower learning rate to let the model converge more slowly.
4. [Less regularization] a penalty tuned too high is deliberately-induced bias.

< Fixes for high variance >

1. [More training data] the only fix that costs nothing in bias. More independent training examples reduce a high-variance model’s tendency to memorize quirks of its current dataset. More validation or test data only makes your performance estimate more reliable.
2. [Regularization] L2 (ridge) shrinks weights, L1 (lasso) zeroes them out; both trade a little bias for a large variance reduction.
3. [Simplify] fewer features, shallower trees, smaller networks.
4. [Averaging] bagging and random forests average many high-variance models into one lower-variance model — the entire point of the ensemble.
5. [Early stopping / dropout] stop before the network starts memorizing, or randomly disable units so it cannot rely on any single path.

< Where the estimate comes from >

• A single train/test split gives a noisy estimate of both. k-fold cross-validation averages over k splits, which is what makes the bias/variance readout stable enough to act on.
• Never tune against the test set — the moment you select a model by test error, that error becomes an optimistic training metric and stops estimating generalization.

< How model families sit on the curve >

Model	Typical position	Main knob
Linear / logistic regression	High bias, low variance	L1/L2 penalty, feature set
Decision tree (unpruned)	Low bias, high variance	Max depth, min samples per leaf
Random forest	Low bias, reduced variance by averaging	Number of trees, feature subsampling
XGBoost	Tunable both ways	Learning rate, depth, regularization terms
Deep networks	Low bias, variance controlled by data volume	Width/depth, dropout, weight decay, early stopping

• Boosting vs bagging — bagging (random forest) attacks variance by averaging independent models; boosting attacks bias by fitting each new model to the previous one's residuals, and therefore needs its own regularization to keep variance in check.

Reference

• deep-learning.ai: Bias and Variance (Andrew Ng)
• The Bias–Variance Tradeoff (Cornell CS4780)
• The Elements of Statistical Learning, ch. 7 — Model Assessment and Selection (Hastie, Tibshirani, Friedman)
• Understanding the Bias-Variance Tradeoff (Scott Fortmann-Roe)
• Bias and variance in scikit-learn (scikit-learn docs)