Regularisation and Optimisation

Training deep neural networks is as much about preventing overfitting and navigating complex loss landscapes as it is about building the right architecture. This lesson covers the key regularisation techniques and optimisation strategies that make deep learning work in practice.

---

Why Regularisation Matters

Deep neural networks have millions of parameters — far more than the number of training examples in most datasets. This gives them enormous capacity to memorise the training data rather than learning generalisable patterns. Regularisation is the set of techniques used to prevent overfitting and improve generalisation.

Symptom	Diagnosis	Solution
Training loss low, validation loss high	Overfitting	Apply regularisation
Both losses high	Underfitting	Increase model capacity or train longer
Both losses low and similar	Good generalisation	Model is well-regularised

---

Dropout

Dropout (Srivastava et al., 2014) randomly sets a fraction of neurons to zero during each training step. This prevents neurons from co-adapting and forces the network to learn redundant representations.

import torch.nn as nn