📝 Activation Functions

Description

An activation function applies a non-linear transformation to a layer's pre-activation value:

$$a = f(x)$$

Without activations, a neural network is only a stack of affine transformations—effectively one linear model, no matter how many layers it has.

Why activation functions are needed

Two affine layers collapse into one:

$$W_2(W_1x+b_1)+b_2=(W_2W_1)x+(W_2b_1+b_2)$$

1. Introduce non-linearity: ReLU, sigmoid, and tanh prevent this collapse, letting a network learn curved decision boundaries and other non-linear patterns.
2. Enable deep representations: Non-linear layers can compose progressively richer features; universal-approximation results show that networks with suitable activations can approximate broad classes of continuous functions.
3. Support gradient flow: Backpropagation applies the chain rule through each activation. Useful derivatives help learning; saturated sigmoid or tanh units can have derivatives near zero.
4. Output Range Normalization: Sigmoid maps one output to $(0, 1)$ for binary probabilities, softmax produces a distribution over classes, and tanh bounds values to $(-1, 1)$.
5. Create sparse activations with ReLU: ReLU turns negative inputs into zero; this can produce sparse features and can be cheaper when an implementation exploits the zeros. Leaky ReLU retains a small negative slope to reduce dead units.

Summary metaphor: Imagine driving a car 🚗. Without an activation function, your steering wheel is bolted straight ahead—you can only go in a straight line. Activation functions are the steering mechanism that lets you turn left, right, and navigate the winding roads of real-world data!