Some one-dimensional datasets are not linearly separable — no single threshold can separate the two classes. The solution is to create a second feature from the original data, mapping the points into two dimensions where they are linearly separable.
Example:
- Class $+1$: ${-3, 3}$ (far from zero)
- Class $-1$: ${-1, 0, 1}$ (close to zero)
No single threshold $t$ works: for any $t$, some class $+1$ points are on the wrong side.
Create a second feature $x_2 = x_1^2$ (the square of the original feature), and map each point $(x_1) \rightarrow (x_1, x_1^2)$:
| $x_1$ | Class | $x_2 = x_1^2$ | 2D point |
|---|---|---|---|
| $-3$ | $+1$ | $9$ | $(-3, 9)$ |
| $3$ | $+1$ | $9$ | $(3, 9)$ |
| $-1$ | $-1$ | $1$ | $(-1, 1)$ |
| $0$ | $-1$ | $0$ | $(0, 0)$ |
| $1$ | $-1$ | $1$ | $(1, 1)$ |
In 2D: class $+1$ points have large $x_2$ (high up), class $-1$ points have small $x_2$ (low down). A horizontal line $x_2 = c$ separates them.
Support vectors (closest across classes): $(3, 9)$ or $(-3, 9)$ from class $+1$, and $(1, 1)$ or $(-1, 1)$ from class $-1$.
Decision boundary $x_2 = 5$ means $x_1^2 = 5$, so $|x_1| = \sqrt{5} \approx 2.24$.
Classification rule in original 1D:
- If $|x_1| > \sqrt{5}$: predict $+1$
- If $|x_1| \leq \sqrt{5}$: predict $-1$
This is a non-linear decision boundary in 1D, found using a linear SVM in 2D.
KEY TAKEAWAY: Mapping from 1D to 2D with $x_2 = x_1^2$ transforms a non-linearly separable problem into a linearly separable one. The linear boundary in 2D corresponds to a non-linear boundary in 1D.
This technique generalises to the kernel trick: map features to a higher-dimensional space where linear separation is possible. The decision boundary in the original space may be curved (parabola, circle, etc.).
| Transformation | New feature | When useful |
|---|---|---|
| Quadratic | $x_2 = x_1^2$ | Data symmetric around zero, classes at different distances from 0 |
| Absolute value | $x_2 = | x_1 |
| Exponential | $x_2 = e^{x_1}$ | Exponential growth patterns |
EXAM TIP: For VCAA: given a 1D dataset that is not linearly separable, apply $x_2 = x_1^2$ to create 2D data, find the SVM boundary in 2D (midpoint between support vectors), then interpret back in 1D.
COMMON MISTAKE: After finding the decision boundary in 2D, you must transform it back to the original 1D space. If the 2D boundary is $x_2 = 5$, the 1D rule is $x_1^2 = 5$, not $x_1 = 5$.
VCAA FOCUS: Know why $x_2 = x_1^2$ is useful, how to apply the transformation, find the 2D boundary, and interpret it in 1D.
