📝 Mathematical Foundations

Mathematical foundations are the tools used to describe data, models, learning, and uncertainty in machine learning.

Vector norms

For a weight vector $w$, $\lVert w\rVert$ means its norm: a single number that measures the vector's size. The subscript specifies which kind of size to use.

For example, if $w=[3,4]$, its L2 norm is its usual Euclidean length:

$$\lVert w\rVert_2 = \sqrt{3^2+4^2} = 5$$

Notation	Meaning	Common ML use
$\lVert w\rVert_1$	Sum of absolute values: $\sum_j \lvert w_j\rvert$	L1 regularization; can encourage sparse weights
$\lVert w\rVert_2$	Euclidean length: $\sqrt{\sum_j w_j^2}$	Measures the overall size of a weight vector
$\lVert w\rVert_2^2$	Squared L2 norm: $\sum_j w_j^2$	L2 regularization / weight decay

Without a subscript, the intended norm must come from context. In regularization, $\lVert w\rVert_2$ is the square root of the sum of squared weights, while $\lVert w\rVert_2^2$ is that sum itself.

Area	Why it matters in ML
Probability & Statistics	Describes uncertainty, data distributions, likelihoods, and model confidence
Linear Algebra	Represents data and model parameters with vectors, matrices, and tensors
Calculus & Optimization	Provides gradients and optimization methods for training models
Information Theory	Explains losses such as cross-entropy and measures distribution differences