📝 Sigmoid

Description

< What is it? >

The sigmoid function, written $\sigma$, squeezes any real-valued logit into the range $(0, 1)$:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

It is commonly used at the output of a binary classifier, where its result can be interpreted as the model's probability for the positive class.

Key points

• Derivative: $\sigma'(z) = \sigma(z)\bigl(1-\sigma(z)\bigr)$
• Not usually a hidden-layer default: for large positive or negative inputs it saturates, making its derivative near zero and slowing gradient flow; ReLU or GELU is usually preferred
• Binary vs. multiclass: use sigmoid for independent binary labels; use softmax for one choice among mutually exclusive classes
• Numerical stability: during training, give logits directly to a combined binary-cross-entropy loss when available instead of computing sigmoid separately

Q&A

< Why use sigmoid, not y = x? >

Q: Why use sigmoid instead of a linear function $y = x$?
A: Exploding gradients is the primary downside. Using $y=x$ (a linear classifier) works in theory but fails in practice:

• Downsides:
  • Exploding gradients: To minimize loss, the model pushes class-1 scores to +∞ and class-0 scores to -∞. This creates massive gradients, causing weight updates to balloon and training to crash (NaN).
  • Brittle thresholds: The decision threshold depends on the arbitrary scale of your weights. Sigmoid naturally fixes the boundary at 0.5; y=x forces you to manually re-tune the threshold for every run.
  • Outlier sensitivity: A single outlier scoring 100 creates a linearly massive gradient, warping the whole model. Sigmoid saturates, making 100 and 2 produce similar gradients, so outliers don't dominate.
  • Crucial nuance: We do output y=x in practice—they're called logits. We don't apply sigmoid manually; instead, we feed them into a loss function like BCEWithLogitsLoss, which applies the sigmoid inside the loss calculation for numerical stability. This ensures we still compare probabilities to labels, but without the gradient chaos.

< Max derivative >

Q: What is the maximum derivative of the sigmoid function?
A: what is the max slope of sigmoid function and why?

For the standard sigmoid function

$$\sigma(x)=\frac{1}{1+e^{-x}}$$

its derivative is

$$\sigma'(x)=\sigma(x)\bigl(1-\sigma(x)\bigr)$$

Let $y=\sigma(x)$, where $0<y<1$. Then

$$\sigma'(x)=y(1-y)=\frac14-\left(y-\frac12\right)^2$$

This is maximized when $y=\frac12$, which occurs at $x=0$. Therefore,

$$\boxed{\max_x \sigma'(x)=\frac14}$$

So the sigmoid’s maximum slope is 0.25 at $x=0$.

For a scaled sigmoid $\sigma(kx)$, the maximum slope becomes $k/4$. This bound also helps explain vanishing gradients: away from zero, the sigmoid derivative approaches $0$, and even at its steepest it is only $0.25$.

< what is the slope when x is 2 or -2? >

Q: What is the slope (derivative / gradient) of the sigmoid function when x is 2 or -2?
A: The sigmoid derivative is

$$\sigma'(x)=\sigma(x)\bigl(1-\sigma(x)\bigr)$$

At $x=2$:

$$\sigma'(2)\approx 0.8808(1-0.8808) =0.8808\times0.1192 \approx \boxed{0.1050}$$

At $x=-2$:

$$\sigma'(-2)\approx 0.1192(1-0.1192) =0.1192\times0.8808 \approx \boxed{0.1050}$$

The slopes are equal because the sigmoid’s derivative is symmetric around (x=0):

$$\boxed{\sigma'(2)=\sigma'(-2)\approx0.105}$$

Related ideas

• Activation Functions
• ReLU Family
• Vanishing & Exploding Gradients