Neural networks are a class of machine learning models inspired by the structure of biological brains. They consist of layers of interconnected units (neurons) that learn to perform classification and other tasks by adjusting connection weights during training.
| Biological Neuron | Artificial Neuron |
|---|---|
| Dendrites | Input connections |
| Cell body | Weighted sum + activation function |
| Axon | Output signal |
| Synapse | Edge weight |
Each artificial neuron computes:
$$z = \sum_{i} w_i x_i + b$$
$$a = \sigma(z)$$
Where:
- $w_i$: input weights
- $b$: bias
- $\sigma$: activation function
- $z$: pre-activation (weighted sum)
- $a$: post-activation (output)
Common activation functions:
| Function | Formula | Range |
|---|---|---|
| Step | $\mathbf{1}[z > 0]$ | ${0, 1}$ |
| Sigmoid | $\frac{1}{1+e^{-z}}$ | $(0, 1)$ |
| ReLU | $\max(0, z)$ | $[0, \infty)$ |
A single perceptron can only solve linearly separable problems (e.g. AND, OR but NOT XOR). By stacking neurons in layers, networks learn non-linear functions.
Why multiple layers? Without non-linear activation functions, stacking layers collapses to a single linear transformation. With them, multi-layer networks are universal function approximators for sufficiently large architectures.
KEY TAKEAWAY: Neural networks learn by adjusting edge weights to minimise prediction error. Multiple layers with non-linear activations enable learning of complex, non-linear patterns.
Strengths:
- Handles complex, non-linear patterns
- Automatic feature learning
- Works with diverse data types (images, text, numbers)
Weaknesses:
- Requires large training datasets
- Computationally expensive
- Difficult to interpret (black box)
- Prone to overfitting without regularisation
Neural networks have gone through cycles of interest and neglect:
- 1943: McCulloch-Pitts neuron
- 1957: Rosenblatt’s Perceptron
- 1969: Minsky and Papert prove limits of single-layer perceptrons
- 1986: Backpropagation popularised by Rumelhart, Hinton, Williams
- 2012: AlexNet dominates ImageNet; deep learning era begins
EXAM TIP: Know the vocabulary: neuron, weight, bias, activation function, layer. Understand why multi-layer networks are more powerful than single-layer perceptrons (non-linear activation functions enable learning non-linear boundaries).
VCAA FOCUS: Know the general structure of neural networks, what neurons compute, and the role of activation functions. Understand the training objective (minimise error by adjusting weights).
