📝 Confidence Calibration

Description

< What is it? >

• [The idea]

  • A model is calibrated when its confidence matches its empirical accuracy. Of all the predictions it makes with confidence 0.8, about 80% should turn out correct — no more, no less.

  • Formally, for predicted probability $\hat p$ and true label $y$:

    $$P\left(\hat y = y \mid \hat p = p\right) = p \qquad \forall\, p \in [0,1]$$

  • Calibration says nothing about whether the model is good. A weather model that predicts "30% rain" every single day, in a climate where it rains 30% of days, is perfectly calibrated and completely useless. Calibration is a property of the probabilities, not of the decisions.

  • Model predicts 50% heads up of coin thrown 🪙 → ✅ calibrated

• [Calibration vs discrimination — why it needs its own metric]

  • Discrimination is the ability to rank: do positives score higher than negatives? That is what accuracy and ROC-AUC measure. (ROC: Receiver Operating Characteristic)

  • AUC (Area Under The Curve) is invariant to any monotonic transform of the scores. Square every probability, or push them all toward 0 and 1, and the ranking — and therefore the AUC — is unchanged, while calibration is destroyed.

  • So a model can be an excellent ranker and wildly overconfident at the same time. Neither accuracy nor AUC will show it. This is the whole reason calibration is measured separately.

    two models, identical AUC = 0.90
    
    A:  p̂ = 0.55, 0.60, 0.65 …   underconfident, well ranked
    B:  p̂ = 0.99, 0.99, 0.99 …   overconfident, well ranked
                                  ↑ same ranking, same AUC, very different ECE

• [When it matters]
  • Any time the probability is used as a number rather than as a rank: expected-value decisions, cost-sensitive thresholds ("treat if risk > 12%"), abstention and hand-off to a human, combining a model's output with another system's, or reporting risk to a person who will act on it.
  • It matters much less for pure ranking tasks — search results, recommendation ordering — where only the sort matters.
• [Why modern networks are miscalibrated]
  • Shallow models of the 1990s and 2000s were close to calibrated out of the box. Modern deep networks are systematically overconfident, and increasing depth, width and capacity makes it worse.
  • The mechanism: after accuracy plateaus, training continues to reduce NLL by pushing the softmax outputs toward 1.0 on examples that are already correct. The model keeps getting more confident without getting more accurate — the classic overfitting shape, seen in the confidence rather than the error rate.

Key points

< Best ways to improve calibration >

1. Use a separate calibration dataset

   Split data into:

   • Training set: learn model parameters
   • Calibration / validation set: learn the calibration mapping
   • Test set: evaluate final calibration

   Never fit the calibration mapping on the training set—it will usually appear unrealistically good.

2. Apply post-training calibration

   2.1 Temperature scaling is usually the first method to try for neural-network classification:

   $$p_i=\operatorname{softmax}\left(\frac{z_i}{T}\right)$$

   Here $z_i$ is the logit for class $i$, and $T$ is learned on the calibration set.

   • $T>1$: reduces overconfidence
   • $T<1$: increases confidence
   • Preserves class ordering and argmax predictions
   • Simple and resistant to overfitting
     
     

   2.2 Platt scaling

   For binary classification, use Platt scaling:

   • Core idea: Apply a sigmoid transformation to the output of an existing model, such as its log-odds or decision score, to recalibrate the predicted probabilities without discarding or retraining the original model.

   • Procedure:

     1. Train the original model, such as an SVM or random forest, and obtain its output score (f(x)). This is usually a decision score or log-odds value.
     2. Set the calibration objective so that $P(y=1\mid f(x))$ matches the observed positive rate. Fit the following transformation by minimizing a loss function such as log loss:
     $$P(y=1\mid f(x)) = \operatorname{sigmoid}(w f(x)+b)$$

     where (w) and (b) are parameters learned from a separate calibration dataset.

     3. Apply the learned sigmoid function to the original model’s output to obtain calibrated probabilities.

   • When to use it: Platt scaling works well when the original model’s score has an approximately sigmoid-shaped relationship with the true probability. It is simple to implement and generally produces stable results.

     

   • Why it complements—not replaces—the original model

     Platt scaling takes the original model’s score as its input:

     $$x \xrightarrow{\text{original model}} f(x) \xrightarrow{\text{Platt scaling}} \sigma(wf(x)+b)$$
     • The original model learns complex relationships between the features $x$ and the outcome. (Let's say the feature could even have some text/embedding, which sigmoid does not understand!!!)
     • The sigmoid calibrator converts the model’s score $f(x)$ into a more accurate probability.

     For example, a random forest may produce $f(x)=2.3$, which Platt scaling converts to $P(y=1\mid x)=\sigma(2.3w+b)$.

     Applying a sigmoid directly to $x$ is simply logistic regression. A linear model may underfit nonlinear feature interactions that a neural network, random forest, or boosted-tree model can learn.

     Why calibrate a model that already uses sigmoid? A sigmoid output can still be miscalibrated because of overfitting, class imbalance, noisy labels, negative sampling, regularization, distribution changes, or optimizing classification accuracy rather than probability quality. For example, a model may predict $0.9$ for a group in which only 70% of examples are positive; calibration aims to map $0.9$ closer to $0.7$. And we might not be able to post-train the model, since the model could be from another team/company, we only do tool call/api call of it, and get value. So we need to calibrate the output of the model, instead of retraining it.

     At serving time, both computations are still required:

     score = original_model.predict_score(x)
     probability = sigmoid(w * score + b)

     If the original model is already well calibrated on unseen data, post-training calibration is unnecessary. Check this with a reliability diagram, Brier score, log loss, and expected calibration error (ECE).

   Other options include:

   • Isotonic regression: flexible, but needs substantial calibration data
   • Beta calibration: useful when the original output is already a probability
   • Vector / Dirichlet scaling: class-specific multiclass calibration

3. Train with an appropriate loss

   Use a proper probability loss such as binary cross-entropy:

   $$L=-\frac{1}{m}\sum_i \left[ y_i\log p_i+(1-y_i)\log(1-p_i) \right]$$

   You can also monitor:

   • Log loss
   • Brier score
   • Calibration error

   Be careful with focal loss, aggressive class weighting, and some forms of label smoothing. They may improve ranking or recall while making raw probabilities less calibrated.

4. Correct sampling and class imbalance

   Suppose your training data down-samples negative examples, which changes the apparent positive rate. For example:

   • Real booking rate: $1\%$
   • Training data after down-sampling: $20\%$ positive
   • Model output: usually not a valid booking probability without correction

   Use one of:

   • Sample weights during training
   • Prior-probability correction
   • Calibration data with the real production distribution

   This is especially important in recommendation and conversion models.

5. Reduce model uncertainty and overfitting

   Helpful techniques include:

   • Early stopping based on validation log loss
   • Weight decay
   • More representative training data
   • Better label quality
   • Deep ensembles
   • Bayesian methods or Monte Carlo dropout

   Model ensembles often improve both predictive accuracy and calibration by averaging unstable predictions.

6. Calibrate for the correct population

   Calibration can differ across:

   • Countries and markets
   • New versus returning users
   • Device types
   • Traffic sources
   • Listing categories
   • Search positions
   • Time periods

   You can use separate segment calibrators when there is enough data. For small segments, use a global calibrator with hierarchical or shrinkage-based adjustments.

7. Handle exposure and position bias first

   Suppose a ranking model trained from clicks sees labels only for displayed items. Top-ranked items receive more clicks partly because they are more visible.

   Possible corrections include:

   • Small randomized-position experiments
   • Inverse propensity weighting
   • Model position effects separately during training rather than use raw position as a relevance feature at serving
   • Doubly robust estimators

   Calibration cannot by itself remove position bias. First correct the biased labels or exposure process, then calibrate the resulting probabilities.

< How to evaluate calibration >

Use a reliability diagram:

1. Put predictions into probability bins.
2. For each bin, compare average predicted probability with observed frequency.
3. A perfectly calibrated model follows the diagonal.

Reliability diagram showing systematic overconfidence in the uncalibrated model. Data points falling below the diagonal indicate the model claims higher confidence than its actual accuracy warrants, with points in the high-confidence region showing the largest deviations from ideal calibration.

Also measure:

$$\mathrm{Brier} = \frac{1}{m}\sum_i(p_i-y_i)^2$$

and:

• Expected Calibration Error (ECE)
• Adaptive ECE
• Log loss
• Calibration slope and intercept
• Metrics broken down by important segments

< A practical workflow >

1. Train the model with cross-entropy.
2. Preserve an unbiased, production-like calibration set.
3. Plot a reliability diagram.
4. Try temperature or Platt scaling first.
5. Compare log loss, Brier score, and ECE on an untouched test set.
6. Monitor calibration by segment and recalibrate when traffic or base rates drift.

For a booking model, calibrate the exact probability you intend to use—for example:

$$P(\text{booking}\mid\text{impression})$$

rather than assuming that a calibrated click probability is also a calibrated booking probability.

Crash course

Reliability diagram & ECE

Model Calibration - Brier Score Explained

PLATT’S CALIBRATION SCALING – LEC 456

Platt’s Calibration Scaling

Reference

• Calibration in Machine Learning: Confidence, Accuracy & ECE (by Michael Brenndoerfer)
• On Calibration of Modern Neural Networks — the overconfidence result and temperature scaling (Guo et al., 2017)
• Predicting Good Probabilities With Supervised Learning (Niculescu-Mizil & Caruana, 2005)
• Measuring Calibration in Deep Learning — why ECE's binning matters (Nixon et al., 2019)
• Can You Trust Your Model's Uncertainty? — calibration under distribution shift (Ovadia et al., 2019)
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (Lakshminarayanan et al., 2017)
• Language Models (Mostly) Know What They Know — calibration and self-evaluation in LLMs (Kadavath et al., 2022)
• Probability calibration — reliability diagrams, Platt, isotonic (scikit-learn docs)
• Platt Scaling for Model Calibration: A Visual Guide (Avi Chawla)
• Platt Scaling & Calibration (Anant Mehta)