📝 Data Normalization

Description

< What is it? >

• [The idea]
  • Normalization (feature scaling) rewrites each input feature onto a common scale before training. It is a preprocessing step, not a model choice: nothing about what the model can express changes, only the numerical conditioning of the problem it is handed.
  • Raw features arrive in whatever units the world used — age in years (0–100), income in dollars (0–500,000), a ratio in [0, 1]. Those units are arbitrary, but most learning algorithms are not invariant to them, so the units silently become weights.
• [Why it matters]
  • Gradient descent geometry. With features on wildly different scales the loss surface is a long, narrow valley. The gradient points across the valley rather than down it, so training zig-zags and the learning rate is capped by the steepest direction. Rescaling makes the contours closer to circular, and the same optimizer converges in far fewer steps.
  • Distance is meaningless without it. kNN, k-means, SVMs with an RBF kernel and PCA all compute distances or variances across features. A feature measured in dollars dominates one measured in ratios purely because its numbers are bigger — it hijacks the metric.
  • Penalties are scale-sensitive. An L1/L2 penalty (see Regularization) charges by weight magnitude, and the weight a feature needs is inversely proportional to its scale. Without scaling, the penalty punishes small-unit features hardest, for no principled reason.
  • Saturating activations. Large inputs push sigmoid/tanh units into their flat regions where gradients vanish, so the layer stops learning.
• [Where it does not matter]
  • Decision trees, random forests and XGBoost split on thresholds within a single feature at a time. Any monotonic rescaling produces the same splits, so scaling changes nothing but wasted time.
  • Ordinary least squares linear regression solved in closed form is also scale-invariant in its predictions — but not in its conditioning, its coefficient interpretability, or the moment you add a penalty.

< The common transforms >

Standardization (z-score)   x' = (x − μ) / σ            → mean 0, std 1, unbounded
Min-max normalization       x' = (x − min) / (max − min) → squeezed into [0, 1]
Robust scaling              x' = (x − median) / IQR      → outlier-resistant
Max-abs scaling             x' = x / max(|x|)            → [−1, 1], preserves zeros/sparsity
Unit-vector (L2) norm       x' = x / ‖x‖₂                → each *row* has length 1

• The first four are per-column (fit one statistic per feature, across rows). Unit-vector normalization is per-row and answers a different question: it makes direction matter and magnitude not — standard for TF-IDF vectors and embedding similarity.
• Names are used loosely in the wild. "Normalization" often means min-max specifically, and often means scaling in general; when it matters, say which formula you mean.

Key points

< The rule that breaks pipelines: fit on train only >

1. [Fit the scaler on the training split] compute μ, σ, min, max from training rows only.
2. [Apply those same numbers to validation, test and production] never re-fit. Re-fitting on the test set leaks its distribution into preprocessing and inflates your score — the model has seen something it will not have at inference.
3. [Inside cross-validation, scale inside each fold] scaling the whole dataset before k-fold leaks every fold's statistics into every other fold. Wrap the scaler and the model in a single pipeline object (sklearn.pipeline.Pipeline) so the framework refits per fold for you.
4. [Ship the fitted scaler with the model] the μ/σ vector is part of the artifact. A model deployed without its exact training-time scaler is silently wrong.

< Choosing a transform >

Transform	Output	Use when	Avoid when
Standardization	Mean 0, std 1	Default. Linear/logistic models, SVM, PCA, neural nets	Data is very heavy-tailed (μ, σ are themselves distorted)
Min-max	[0, 1]	You need a bounded range — image pixels, some NN inputs, distance metrics that assume a box	Outliers exist; a single extreme value crushes everything else toward 0
Robust	Median 0, IQR 1	Outliers you want to keep but not be dominated by	Data is clean and roughly symmetric — no benefit
Max-abs	[−1, 1]	Sparse matrices — it never shifts, so zeros stay zeros	Dense data where centering would help
Unit-vector	Row norm 1	Text/TF-IDF, embeddings, cosine similarity	Feature magnitude carries real signal

< The awkward cases >

• Skewed distributions. Scaling moves and stretches a distribution; it does not reshape it. For heavy right skew (income, counts, durations) apply log1p, Box-Cox or Yeo-Johnson first, then standardize. Quantile transforms map to a uniform or normal shape directly.
• Sparse data. Never mean-center a sparse matrix — subtracting μ makes every zero nonzero and can explode a 1 GB matrix into 100 GB. Use max-abs, or standardize with_mean=False.
• One-hot / binary columns. Already on a comparable scale; scaling them is usually unnecessary and makes coefficients harder to read. Leave them, and scale only the continuous columns (ColumnTransformer).
• Outliers. Clip or winsorize before scaling if the extremes are errors; use robust scaling if they are real.
• The target. For regression, scaling y can help optimization but every prediction must be inverse-transformed back. Metrics computed on scaled targets are not comparable to anything.
• Train/serve skew. Categorical encodings, missing-value imputation and scaling must happen identically in both paths. This is the single most common source of "the model performed worse in production".

< Normalization inside the network >

Input scaling is preprocessing; these are layers that renormalize activations during the forward pass, and they solve a related but distinct problem — keeping the distribution of each layer's inputs stable as the weights below it change.

Layer	Normalizes over	Typical home
Batch norm	Each feature, across the minibatch	CNNs / vision. Batch-size dependent; train and inference behave differently
Layer norm	All features, within one example	Transformers, RNNs. Batch-size independent
RMSNorm	Same as layer norm, scale only (no mean subtraction)	Modern LLMs — cheaper, works as well in practice

• Internal normalization reduces but does not remove the need to scale inputs: the first layer still receives raw features, and its gradients still see the raw geometry.
• Vision convention: divide pixels by 255, then subtract per-channel mean and divide by per-channel std using the statistics of the pretraining dataset — using the wrong published constants is a classic silent accuracy loss when fine-tuning.

< Mistakes to check for >

Mistake	Symptom
Fit scaler on the full dataset before splitting	Validation score better than production, unexplained
Scaler refit at inference on one batch	Predictions swing with batch composition
Forgot to inverse-transform predictions	Regression outputs near 0, "the model predicts nothing"
Scaled the one-hot columns and interpreted coefficients	Coefficient sizes no longer comparable
Min-max on data with an outlier	Almost all values collapse into a tiny sliver of [0, 1]
Centered a sparse matrix	Memory blow-up, or a training job that never starts

Reference

• Preprocessing data — scalers, transformers, and when to use each (scikit-learn docs)
• Common pitfalls: data leakage in preprocessing (scikit-learn docs)
• Normalizing Inputs (Andrew Ng, deeplearning.ai)
• Efficient BackProp, §4.3 — why centered, decorrelated inputs speed up learning (LeCun et al.)