📝 Neural Networks

Description

< What is it? >

A neural network is a function built by composing many small units — neurons — arranged in layers. Each neuron does the same two things: combine its inputs linearly, then pass the result through a non-linear activation. Stacking those layers lets the network approximate mappings that no single linear model can express.

        w₁
   x₁ ────────┐
        w₂    │
   x₂ ────────┼──▶  z = Σ wᵢxᵢ + b  ──▶  a = f(z)  ──▶  to the next layer
        w₃    │        (linear)            (activation)
   x₃ ────────┘
                     b (bias)

Bionics

1 hidden layer neural network

Training adjusts the weights and biases; the structure and the choice of activation stay fixed.

Key points

< Neurons and connections >

A neuron holds no data of its own — it is a rule for turning a vector of inputs into one number. A connection carries the output of one neuron to the input of another, and every connection has a weight attached to it. In a fully connected layer, every neuron in layer $l-1$ connects to every neuron in layer $l$, so a layer with $n$ inputs and $m$ outputs has $n \times m$ connections.

< Weights and biases >

The weight $w_i$ on a connection says how strongly that input pushes the neuron's output up or down; a weight near zero makes the input irrelevant. The bias $b$ is an input-independent offset that shifts the neuron's threshold, letting it fire even when every input is zero.

Together they are the network's trainable parameters. A layer collects them into a matrix $W$ and a vector $b$:

$$W \in \mathbb{R}^{m \times n}, \quad b \in \mathbb{R}^{m}$$

Their starting values matter — see Weight Initialization.

< Propagation function: the linear part >

The propagation function is what aggregates a neuron's incoming signals into a single pre-activation value $z$. In practice it is the weighted sum plus bias, which for a whole layer is one matrix multiply:

$$z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}$$

where $a^{(l-1)}$ is the previous layer's output and $a^{(0)} = x$, the input. This is a linear function of the inputs (affine, once the bias is included): scaling an input scales its contribution proportionally, and contributions add.

< Activation function: the non-linear part >

The activation function $f$ maps the pre-activation to the neuron's output:

$$a^{(l)} = f\!\left(z^{(l)}\right)$$

It is applied element-wise, so it changes each value's shape but not the layer's dimensions.

Activation	Formula	Typically used for
ReLU	$\max(0, z)$	Default for hidden layers — cheap, and its gradient does not saturate for $z > 0$
GELU	$z \cdot \Phi(z)$	Hidden layers in transformers
Sigmoid	$1 / (1 + e^{-z})$	A single output squashed to $(0, 1)$ for binary classification
Tanh	$(e^{z} - e^{-z}) / (e^{z} + e^{-z})$	Outputs centred on zero, common in older recurrent networks
Softmax	$e^{z_i} / \sum_j e^{z_j}$	An output layer producing a distribution over classes

< Why the non-linearity is required >

Without $f$, depth buys nothing. Two stacked linear layers collapse into one:

$$W_2(W_1 x + b_1) + b_2 = (W_2 W_1)x + (W_2 b_1 + b_2)$$

which is just another affine map. The activation is what makes a deep network more expressive than a single layer. Its derivative also decides how well gradients survive the trip back through many layers — see Vanishing & Exploding Gradients.

< In PyTorch >

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 128),  # W and b: the propagation function
    nn.ReLU(),            # f: the activation function
    nn.Linear(128, 10),   # output layer — logits, softmax folded into the loss
)

nn.Linear(n, m) allocates the weight matrix and bias vector; nn.ReLU() has no parameters at all.

< Related ideas >

• Feed Forward Network is the simplest arrangement of these layers.
• Backpropagation computes how the loss changes with each weight and bias.
• Gradient Descent uses those gradients to update them.

Reference

• Neural Networks and Deep Learning — Michael Nielsen
• Deep Learning, Chapter 6: Deep Feedforward Networks — Goodfellow, Bengio & Courville
• PyTorch nn.Linear documentation
• Introduction To Neural Networks (geeksforgeeks.org)
• Activation Functions in Neural Networks (geeksforgeeks.org)