📝 Model Drift

Description

Model drift is a meaningful change in a deployed model’s inputs, outputs, or real-world performance because production no longer matches the conditions in which the model was trained.
A useful decomposition is:
$P(X,Y)=P(Y\mid X)P(X)$

Covariate / data drift — $P(X)$ changes: the distribution of input features differs between training and production, while $P(Y\mid X)$ may remain unchanged. For example, users begin searching for longer stays.
Label / prior drift — $P(Y)$ changes: the overall booking rate falls from 5% to 3%.
Concept drift — $P(Y\mid X)$ changes: the relationship between price and booking behavior changes.
Prediction drift — $P(\hat{Y})$ changes: the model starts producing much lower scores. This is usually a warning signal, not the root cause.
Calibration drift: predicted probabilities no longer match observed frequencies; listings predicted at 20% book only 12% of the time.
Feature / pipeline drift: feature computation changes or breaks, such as a failed currency conversion. This is an operational problem, not necessarily population change.

Drift can be sudden, gradual, recurring (for example, seasonal), or segment-specific to a country, device, or user cohort.

Use multiple monitoring layers. A shifted feature distribution alone does not prove that model quality has deteriorated.

Monitor inputs: compare recent production data with a suitable reference period.
- Numerical features: missing rate, mean, variance, quantiles, PSI, KS statistic, or Wasserstein distance.
- Categorical features: category frequencies, unseen-category rate, Jensen–Shannon divergence, or chi-squared tests.
- Embeddings: centroid and norm distributions, Maximum Mean Discrepancy (MMD), or a classifier that distinguishes reference data from current data.
Monitor predictions: labels may arrive late, so watch average scores, score quantiles and variance, positive-decision rate, confidence or entropy, predicted-class mix, and ranking-score distributions.
Monitor performance: track task metrics, probability quality, calibration, and business outcomes over time and by important segments.
- Classification: precision, recall, ROC-AUC, and PR-AUC.
- Probability quality: log loss, Brier score, ECE, and reliability diagrams.
- Regression or ranking: MAE / RMSE or Recall@K / NDCG@K.
Check operations first: look for training-serving feature skew, missing or stale features, schema or unit changes, model-version mismatches, increased defaults, latency or fallback changes, and broken logging or label joins. A pipeline bug can look like concept drift.
Design useful alerts: consider minimum sample size, effect size, persistence, seasonality, historical variability, and business impact—not only statistical significance. Compare holiday traffic with a comparable holiday period.

Feature or pipeline problem: roll back the broken feature or model version, repair the pipeline, use safe defaults or a fallback, and exclude corrupted periods from future training. Do not retrain merely to accommodate a pipeline bug.
Covariate drift: retrain on recent representative data, use a rolling window, reweight or resample affected segments, add explanatory features, or apply domain adaptation.
Label / base-rate drift: recalibrate probabilities, adjust an intercept or decision threshold, correct changed priors, and retrain if the change persists. If ranking remains strong but probabilities are too high, recalibration can be enough.
Concept drift: retrain on recent labeled examples, shorten the retraining interval, add features for the new behavior, revise labels or objectives, or use incremental learning when rapid adaptation is needed.
Severe degradation: roll back to a stable model, reduce traffic, switch to a simpler fallback, run champion–challenger evaluation, and restore traffic gradually.

Retraining is not the automatic response to every alert. First rule out seasonality, delayed labels, and operational defects.

Data Drift vs Model Drift in Machine Learning Causes, Detection & Prevention Explained