📝 Softmax

Description

< What is it? >

Softmax converts a vector of arbitrary real-valued scores, called logits, into a probability distribution over $C$ mutually exclusive classes:

$$\operatorname{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{C}e^{z_j}}$$

Here, $\mathbf{z}=[z_1,z_2,\dots,z_C]$ is the vector of logits. $z_i$ is the logit for the specific class whose probability is being calculated, while $z_j$ refers to every logit as $j$ runs from $1$ to $C$ in the denominator. In other words, $i$ identifies the target class and $j$ is a temporary index used to sum over all classes.

• Example: for $\mathbf{z}=[2,1,0]$, the probability of the first class is

  $$p_1 = \frac{e^{z_1}}{e^{z_1}+e^{z_2}+e^{z_3}} = \frac{e^2}{e^2+e^1+e^0}$$

The output for every class is between $0$ and $1$, and all output probabilities sum to $1$. It is commonly used at the output of a multiclass classifier.

Key points

< From logits to probabilities >

• A logit is an unnormalized model score. Softmax compares every logit with all the others, so raising one class's probability lowers the relative probability of the others.
• Softmax preserves ranking: the class with the largest logit also has the largest probability.
• Example: logits $[2,1,0]$ become approximately $[0.665,0.245,0.090]$. The first class is most likely, and the three probabilities sum to $1$.

< Temperature >

A positive temperature $T$ can make the distribution softer or sharper:

$$p_i = \operatorname{softmax}\left(\frac{\mathbf{z}}{T}\right)_i$$

• $T>1$ produces a flatter, less confident distribution
• $0<T<1$ produces a sharper, more confident distribution
• For any $T>0$, it preserves the predicted class because the logit ordering does not change

< Numerical stability >

Exponentials can overflow for large logits. A stable implementation subtracts the largest logit before applying the exponential:

$$\operatorname{softmax}(\mathbf{z})_i = \frac{e^{z_i-z_{\max}}}{\sum_{j=1}^{C}e^{z_j-z_{\max}}} \qquad z_{\max}=\max_j z_j$$

• Example: suppose an image classifier has logits cat = 1000 and dog = 999. Computing $e^{1000}$ can overflow, but subtracting the maximum changes the logits to $[1000,999]-1000=[0,-1]$. Softmax then safely produces approximately $[0.731,0.269]$ for cat and dog.

This gives exactly the same probabilities. During training for single-label classification, pass logits directly to a numerically stable cross-entropy loss instead of applying softmax manually first.

Comparison

< Softmax vs sigmoid >

	Softmax	Sigmoid
Output	One probability distribution that sums to $1$	Independent values between $0$ and $1$
Best for	One mutually exclusive class	One binary label or multiple independent labels
Example	Choose exactly one: cat, dog, or bird	An image can be a cat, indoors, and sunny at the same time
Typical loss	Multiclass cross-entropy	Binary cross-entropy

Related ideas

• Activation Functions explains why neural networks use non-linear functions.
• Sigmoid is the usual output function for binary or multi-label classification.
• Transformer uses softmax to turn attention scores into attention weights.
• Confidence Calibration uses temperature scaling to adjust predicted probabilities.