Multi-Layer Perceptron Structure

Structure of Multi-Layer Perceptron (MLP) Neural Networks

A multi-layer perceptron (MLP) is a feedforward neural network with at least three layers. Information flows in one direction: input to hidden to output.

Layers

Fully connected (dense): Every neuron in each layer connects to every neuron in the next layer.

Notation

For a network with $d$ inputs, $h$ hidden neurons (one hidden layer), $k$ outputs:

• $W^{(1)}$: weight matrix, shape $h \times d$ (input-to-hidden weights)
• $W^{(2)}$: weight matrix, shape $k \times h$ (hidden-to-output weights)
• $\mathbf{b}^{(1)}$: bias vector, length $h$
• $\mathbf{b}^{(2)}$: bias vector, length $k$

Diagram

Input layer      Hidden layer     Output layer
  x_1  -----\
              \--> h_1 --\
  x_2  ---------> h_2 ----> output y_1
              \--> h_3 --/
  x_3  -----/

Each arrow represents a weight.

Computing a Single Hidden Neuron

For hidden neuron $j$ in the first hidden layer:
$$z_j^{(1)} = \sum_{i=1}^{d} w_{ji}^{(1)} x_i + b_j^{(1)}$$
$$a_j^{(1)} = \sigma\bigl(z_j^{(1)}\bigr)$$

In matrix form for the entire hidden layer:
$$\mathbf{z}^{(1)} = W^{(1)} \mathbf{x} + \mathbf{b}^{(1)}, \qquad \mathbf{a}^{(1)} = \sigma\bigl(\mathbf{z}^{(1)}\bigr)$$

Example: 2 inputs, 2 hidden neurons, 1 output

Weights:
$$W^{(1)} = \begin{pmatrix} 1 & 2 \ -1 & 0 \end{pmatrix}, \quad \mathbf{b}^{(1)} = \begin{pmatrix} 0 \ 1 \end{pmatrix}, \quad W^{(2)} = (1,\; -1), \quad b^{(2)} = 0$$

Input: $\mathbf{x} = (1, 1)$

Hidden pre-activations:
$$z_1 = 1 \cdot 1 + 2 \cdot 1 + 0 = 3, \quad z_2 = -1 \cdot 1 + 0 \cdot 1 + 1 = 0$$

Hidden activations (sigmoid):
$$a_1 = \sigma(3) \approx 0.953, \quad a_2 = \sigma(0) = 0.5$$

Output:
$$z^{(2)} = 1 \cdot 0.953 + (-1) \cdot 0.5 + 0 = 0.453$$
$$\hat{y} = \sigma(0.453) \approx 0.611$$

KEY TAKEAWAY: An MLP processes information by sequentially applying weighted sums and activation functions layer by layer. Each hidden layer learns a higher-level representation of the input.

Role of Activation Functions

Without non-linear activations, any MLP collapses to a single linear transformation regardless of depth. Non-linear activations (sigmoid, ReLU) are what give deep networks their expressive power.

EXAM TIP: VCAA will ask you to compute forward propagation on a small MLP. Practice the full calculation: compute $z$ for each neuron, apply $\sigma$, pass results to the next layer. Show all working.

VCAA FOCUS: Describe the layered structure (input, hidden, output). Explain the role of weights and biases. Know how to perform forward propagation through a small network.