📝 ReLU Family

Description

< What is it? >

ReLU stands for Rectified Linear Unit: it rectifies a linear input by setting negative values to zero while leaving positive values unchanged. The ReLU family keeps this simple, non-saturating positive side. Its functions are commonly used in hidden layers because they are inexpensive and help gradients flow more easily than sigmoid or tanh.

\operatorname{ReLU}(x) = \max(0, x)

Key points

Function	Definition	Main trade-off
ReLU	$\max(0, x)$	Fast and a strong default, but its gradient is zero for negative inputs `Pick ReLU for speed`
Leaky ReLU	$\max(x,α·x)$	Keeps a small negative-side gradient, reducing dead ReLU units. One small leak, the network stays alive !!! `Pick Leaky ReLU when neurons keep dying`
PReLU	Leaky ReLU with learnable $\alpha$	Lets the model choose the negative slope, with a few extra parameters

< ReLU vs. Leaky ReLU >

For ReLU, a neuron that repeatedly receives negative inputs outputs zero and receives zero local gradient. It can stop learning; this is called a dead ReLU.

Leaky ReLU keeps a small slope on the negative side:

\operatorname{LeakyReLU}_{\alpha}(x) = \begin{cases} x & x > 0 \\ \alpha·x & x \leq 0 \end{cases} \qquad \alpha \text{ is often } 0.01

Start with ReLU for most hidden layers. Try Leaky ReLU when many units remain at zero or training benefits from a nonzero negative-side gradient. Pair either with He / Kaiming initialization.

Neural Networks introduces activation functions in a neuron
Vanishing & Exploding Gradients explains why activation derivatives matter

Implementation

import numpy as np

def leaky_relu(x, alpha=0.01):
    """
    Applies the Leaky ReLU activation element-wise.

    Leaky ReLU is defined as:
        f(x) = max(x, alpha * x)

    Args:
        x (np.ndarray): Input values (a 1D vector).
        alpha (float): Negative-slope coefficient.

    Returns:
        np.ndarray: Output vector after applying Leaky ReLU.
    """
    # np.maximum computes the element‑wise maximum of two arrays, which directly matches the definition f(x) = max(x, α·x)
    # np.max is an aggregation function (Finds the single highest value), while np.maximum is an element-wise comparison function
    return np.maximum(x, alpha * x)

# Example inputs — edit these to try another case
x = np.array([-2.0, -1.0, 0.0, 3.0], dtype=np.float64)
alpha = 0.1

# Example Usage
leaky_relu(x, alpha)

Q & A

< Why not sigmoid func? >

Q: Since the ReLU function has the dying ReLU problem, why don't we use the sigmoid function instead?
A: Because Sigmoid suffers from the Vanishing Gradient Problem, which is far worse than Dying ReLU.
A dying ReLU is a real failure mode, but replacing it with sigmoid in a deep hidden layer usually trades a local zero-gradient problem for a broader saturation problem. Sigmoid is still useful at a binary-classification output, where a bounded probability is exactly what we want.

The Vanishing Gradient is Global (Dying ReLU is Local)
- Dying ReLU: Only some neurons die (those whose inputs are negative). The rest of the network (all positive neurons) continues learning just fine.
- Sigmoid: Its derivative has a maximum value of only 0.25 (at exactly 0). For any input greater than ~2 or less than ~-2, the derivative drops to near zero. While the derivative (slope / gradient) of ReLu is 1 for positive inputs, which avoid the vanishing gradient problem.
- The result: In a 10-layer network, you multiply 0.25 * 0.25 * 0.25... ten times → the gradient vanishes to practically zero in the early layers. Every single neuron in the first few layers stops learning—not just a few, but all of them.
Dying ReLU is Easy to Fix; Vanishing Sigmoid is Not
- Fixing Dying ReLU: You literally just implemented the fix in your last coding problem—Leaky ReLU (max(x, α·x)), or use ELU, or simply lower your learning rate. It's a 1-line code change.
- Fixing Sigmoid Vanishing: To train deep networks with Sigmoid, you need meticulous weight initialization (Xavier), Batch Normalization at every layer, and skip connections—and even then, it still struggles beyond 5-10 layers. ReLU allows 1000-layer networks out of the box.
Sigmoid is Computationally Expensive
- ReLU: f(x) = max(0, x) → just a simple comparison and max operation. Extremely fast.
- Sigmoid: f(x) = 1 / (1 + e^(-x)) → requires computing an exponential (e^x) for every single neuron. In a large network with millions of neurons, this slows training significantly.
Sigmoid Kills Sparsity (Efficiency)
- ReLU: Outputs exactly 0 for negative inputs. This naturally creates sparsity (many neurons are inactive). Sparse networks are more computationally efficient because you skip computing those zero activations.
- Sigmoid: Outputs a small positive value (e.g., 0.01 or 0.1) even for negative inputs. Every neuron always "fires" a little bit. This forces the network to do all the math for every single neuron, wasting resources. Worse, these tiny positive values accumulate and mess up the optimization (it becomes harder to find the decision boundary).
Sigmoid is Not Zero-Centered
- Sigmoid outputs are always positive (between 0 and 1).
- When updating weights, the gradient for any given weight is always the same sign as the neuron's output (positive). This causes the gradients to zig-zag (oscillate) during optimization, making convergence slower and less stable. ReLU outputs can be both positive and negative (since inputs can be positive or negative), avoiding this issue.

< The final verdict >

Issue	Dying ReLU	Vanishing Sigmoid
Scope	Affects a few neurons	Affects the entire network
Fix	Easy (Leaky ReLU)	Hard (needs complex tricks)
Training Depth	Allows 1000+ layers	Struggles beyond 5-10 layers
Speed	Very fast	Slow (exponential operations)
Sparsity	Yes (efficient)	No (inefficient)

Metaphor:

ReLU's "dying" problem is like a few soldiers falling asleep in a 1000-person army—inconvenient, but easily solved by giving them coffee (Leaky ReLU).
Sigmoid's vanishing problem is like the soldiers are passing a bottle of water to drink, one by one, and last soldier got an empty bottle, and there's no simple fix. That's why ReLU (and its variants like Leaky ReLU) are the default for deep networks, and Sigmoid is almost exclusively used only at the output layer for binary classification (where we want a probability between 0 and 1).

In practice,

Use ReLU, Leaky ReLU, or GELU for most hidden layers
Use sigmoid for an independent binary or multi-label output layer.

Description​

< What is it? >​

Key points​

< ReLU vs. Leaky ReLU >​

< Related ideas >​

Implementation​

Q & A​

< Why not sigmoid func? >​

< The final verdict >​

Crash course​