📝 Neural Networks

Description

< What is it? >

A neural network is a function built by composing many small units — neurons — arranged in layers. Each neuron does the same two things: combine its inputs linearly, then pass the result through a non-linear activation. Stacking those layers lets the network approximate mappings that no single linear model can express.

        w₁
   x₁ ────────┐
        w₂    │
   x₂ ────────┼──▶  z = Σ wᵢxᵢ + b  ──▶  a = f(z)  ──▶  to the next layer
        w₃    │        (linear)            (activation)
   x₃ ────────┘
                     b (bias)

Bionics

1 hidden layer neural network

Training adjusts the weights and biases; the structure and the choice of activation stay fixed.

Key points

< Input, hidden, and output layers >

A neural network usually has three roles of layers:

input features or embeddings → hidden layers → task-specific output layer

• Input layer: receives numeric features, pixels, or embeddings.
• Hidden layers: repeatedly transform the representation so the model can learn useful intermediate features.
• Output layer: produces the value needed by the task, such as one real number, one binary logit, or one logit per class.

< Neurons and connections >

A neuron holds no data of its own — it is a rule for turning a vector of inputs into one number. A connection carries the output of one neuron to the input of another, and every connection has a weight attached to it. In a fully connected layer, every neuron in layer $l-1$ connects to every neuron in layer $l$, so a layer with $n$ inputs and $m$ outputs has $n \times m$ connections.

< Weights and biases >

The weight $w_i$ on a connection says how strongly that input pushes the neuron's output up or down; a weight near zero makes the input irrelevant. The bias $b$ is an input-independent offset that shifts the neuron's threshold, letting it fire even when every input is zero.

Together they are the network's trainable parameters. A layer collects them into a matrix $W$ and a vector $b$:

$$W \in \mathbb{R}^{m \times n}, \quad b \in \mathbb{R}^{m}$$

For a dense layer with $n$ inputs and $m$ outputs, this is $mn+m$ trainable parameters. CNNs and Transformers use more structured connections or weight sharing, but the same idea—learn parameters that transform representations—still applies.

Their starting values matter — see Weight Initialization.

< Propagation function: the linear part >

The propagation function is what aggregates a neuron's incoming signals into a single pre-activation value $z$. In practice it is the weighted sum plus bias, which for a whole layer is one matrix multiply:

$$z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}$$

where $a^{(l-1)}$ is the previous layer's output and $a^{(0)} = x$, the input. This is a linear function of the inputs (affine, once the bias is included): scaling an input scales its contribution proportionally, and contributions add.

< Batches and tensor shapes >

Models process many examples together as a batch. In PyTorch, nn.Linear(n, m) maps an input of shape (B, n) to an output of shape (B, m), where B is the batch size. More generally, it transforms the final dimension, so (B, T, n) becomes (B, T, m) for a sequence of T token representations.

< Activation function: the non-linear part >

The activation function $f$ maps the pre-activation to the neuron's output:

$$a^{(l)} = f\!\left(z^{(l)}\right)$$

It is applied element-wise, so it changes each value's shape but not the layer's dimensions.

Activation	Formula	Typically used for
ReLU	$\max(0, z)$	Default for hidden layers — cheap, and its gradient does not saturate for $z > 0$
GELU	$z \cdot \Phi(z)$	Hidden layers in transformers
Sigmoid	$1 / (1 + e^{-z})$	A single output squashed to $(0, 1)$ for binary classification
Tanh	$(e^{z} - e^{-z}) / (e^{z} + e^{-z})$	Outputs centred on zero, common in older recurrent networks
Softmax	$e^{z_i} / \sum_j e^{z_j}$	An output layer producing a distribution over classes

< Why the non-linearity is required >

Without $f$, depth buys nothing. Two stacked linear layers collapse into one:

$$W_2(W_1 x + b_1) + b_2 = (W_2 W_1)x + (W_2 b_1 + b_2)$$

which is just another affine map. The activation is what makes a deep network more expressive than a single layer. Its derivative also decides how well gradients survive the trip back through many layers — see Vanishing & Exploding Gradients.

< Output layer and loss >

The output layer and loss function depend on the task:

Task	Network output	Common training loss	Interpretation at inference
Regression	One or more real values	MSE or MAE	Predicted quantity, such as price
Binary classification	One logit	Binary cross-entropy with logits	Apply sigmoid to obtain a positive-class probability
Multiclass classification	One logit per mutually exclusive class	Cross-entropy	Apply softmax to obtain class probabilities

Stable framework losses usually accept logits directly, so do not manually apply sigmoid or softmax before those losses.

< How a network learns >

Training repeatedly follows this loop:

batch → forward pass → predictions/logits → loss
      → backpropagation → gradients → optimizer update

The forward pass produces predictions. The loss measures their error, backpropagation computes how each trainable parameter affected that error, and an optimizer uses gradient descent to update the parameters. At inference time, the learned parameters are fixed and the model performs only the forward pass.

< In PyTorch >

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 128),  # W and b: the propagation function
    nn.ReLU(),            # f: the activation function
    nn.Linear(128, 10),   # output layer — logits, softmax folded into the loss
)

This model maps a batch of 784-dimensional inputs to 10 logits. nn.Linear(n, m) allocates the weight matrix and bias vector; nn.ReLU() has no parameters at all.

Comparison

< Linear model vs neural network >

Aspect	Linear model	Neural network
Function	One affine mapping	Many affine mappings separated by non-linear activations
Intermediate features	Usually engineered outside the model	Learned in hidden layers
Strengths	Simple, fast, interpretable, often strong for small or well-structured data	Can model complex patterns in images, audio, text, and other high-dimensional inputs
Tradeoffs	Cannot represent complex non-linear relationships by itself	Usually needs more data, compute, tuning, and regularization

Related ideas

• Multilayer Perceptron (MLP) combines fully connected layers for prediction and representation learning.
• Embeddings turn discrete inputs such as tokens into vectors for neural networks.
• Feed Forward Network is the simplest arrangement of these layers.
• Activation Functions introduce the non-linearity that makes depth useful.
• Backpropagation computes how the loss changes with each weight and bias.
• Gradient Descent uses those gradients to update them.

Video Tutorial

Reference

• Neural Networks and Deep Learning — Michael Nielsen
• Deep Learning, Chapter 6: Deep Feedforward Networks — Goodfellow, Bengio & Courville
• PyTorch nn.Linear documentation
• Introduction To Neural Networks (geeksforgeeks.org)
• Activation Functions in Neural Networks (geeksforgeeks.org)