What is Deep Learning

Deep learning is a subset of machine learning that uses artificial neural networks with multiple layers to learn hierarchical representations of data. While traditional machine learning often relies on hand-crafted features, deep learning models automatically discover the representations needed for a given task — from raw pixels, text, or audio.

---

A Brief History

• 1943 — Warren McCulloch and Walter Pitts publish a mathematical model of an artificial neuron
• 1958 — Frank Rosenblatt builds the Perceptron, the first trainable neural network
• 1969 — Minsky and Papert publish Perceptrons, highlighting the limitations of single-layer networks and triggering an "AI winter"
• 1986 — Rumelhart, Hinton, and Williams popularise backpropagation, enabling training of multi-layer networks
• 1989 — Yann LeCun applies convolutional neural networks to handwritten digit recognition
• 1997 — Hochreiter and Schmidhuber introduce Long Short-Term Memory (LSTM) networks
• 2006 — Geoffrey Hinton coins the term "deep learning" and demonstrates effective training of deep belief networks
• 2012 — AlexNet wins the ImageNet competition by a large margin, sparking the modern deep learning revolution
• 2014 — Ian Goodfellow introduces Generative Adversarial Networks (GANs)
• 2015 — ResNet achieves superhuman performance on ImageNet with 152 layers
• 2017 — The Transformer architecture is published in Attention Is All You Need
• 2018 — BERT demonstrates the power of pre-trained language models
• 2020 — GPT-3 showcases remarkable few-shot learning capabilities
• 2022 — Diffusion models (DALL-E 2, Stable Diffusion) revolutionise image generation
• Today — Deep learning powers applications from autonomous driving and medical imaging to large language models and robotics

---

Deep Learning vs Traditional Machine Learning

Aspect	Traditional ML	Deep Learning
Feature engineering	Manual — domain experts design features	Automatic — the network learns features from raw data
Data requirements	Works well with smaller datasets	Typically requires large datasets to shine
Compute requirements	CPU is often sufficient	GPU/TPU acceleration is essential
Model interpretability	Often more interpretable (e.g., decision trees)	Often treated as a "black box"
Performance ceiling	Plateaus with more data	Continues to improve with more data and larger models
Best for	Tabular data, small/medium datasets	Images, text, audio, video, large-scale data

---

How Deep Learning Works — A High-Level View

A deep learning model is composed of layers of artificial neurons. Each layer transforms its input and passes the result to the next layer, progressively extracting higher-level features.

Example: Image Recognition

1. Input layer — receives raw pixel values
2. Early hidden layers — learn edges and textures
3. Middle hidden layers — learn parts and patterns (e.g., eyes, wheels)
4. Later hidden layers — learn high-level concepts (e.g., faces, cars)
5. Output layer — produces the final prediction (e.g., "cat" or "dog")

Input → [Edges] → [Textures] → [Parts] → [Objects] → Output
         Layer 1    Layer 2      Layer 3    Layer 4

Tip: The word "deep" in deep learning refers to the number of layers in the network. A network with many layers is called a "deep" network.

---

Key Terminology

Term	Definition
Neuron / Node	A computational unit that applies a weighted sum and an activation function to its inputs
Layer	A collection of neurons that operate at the same level of abstraction
Weights	Learnable parameters that determine the strength of connections between neurons
Bias	An additional learnable parameter added to the weighted sum before the activation function
Activation Function	A non-linear function (e.g., ReLU, Sigmoid) applied to a neuron's output
Loss Function	A function that measures how far the model's predictions are from the true values
Backpropagation	The algorithm that computes gradients of the loss with respect to each weight
Gradient Descent	The optimisation algorithm that updates weights to minimise the loss
Epoch	One complete pass through the entire training dataset
Batch Size	The number of training examples processed before updating weights
Learning Rate	A hyperparameter that controls how much weights are adjusted at each step

---

Types of Deep Learning Architectures

Architecture	Best For	Key Idea
Feedforward (MLP)	Tabular data, simple tasks	Layers connected in sequence; no cycles
Convolutional (CNN)	Images, spatial data	Learns local patterns using sliding filters
Recurrent (RNN/LSTM/GRU)	Sequences, time series, text	Maintains hidden state across time steps
Transformer	NLP, vision, multimodal tasks	Self-attention mechanism processes all positions in parallel
Autoencoder	Dimensionality reduction, denoising	Learns a compressed representation and reconstructs the input
GAN	Image generation, data augmentation	Two networks (generator and discriminator) compete
Diffusion Model	Image and audio generation	Learns to reverse a noise-adding process

---

Python Ecosystem for Deep Learning

Library	Description
PyTorch	Flexible deep learning framework by Meta; dominant in research
TensorFlow	Production-focused deep learning framework by Google
Keras	High-level API (integrated into TensorFlow) for rapid prototyping
JAX	High-performance numerical computing with automatic differentiation (Google)
Hugging Face Transformers	Pre-trained models for NLP, vision, and audio
torchvision / torchaudio	PyTorch extensions for computer vision and audio
NumPy	Fundamental numerical computing library
Matplotlib	Plotting and visualisation

A First Deep Learning Example with PyTorch

import torch
import torch.nn as nn

# Define a simple feedforward network
class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 128)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.fc2(x)
        return x

model = SimpleNet()
print(model)

---

Real-World Applications of Deep Learning

Computer Vision

• Image classification and object detection
• Medical image analysis (X-rays, MRIs, pathology slides)
• Autonomous driving (pedestrian detection, lane keeping)

Natural Language Processing

• Machine translation (Google Translate)
• Chatbots and virtual assistants
• Sentiment analysis and text summarisation

Audio and Speech

• Speech recognition (Siri, Alexa)
• Music generation and audio enhancement
• Speaker identification

Science and Healthcare

• Protein structure prediction (AlphaFold)
• Drug discovery and molecular design
• Climate modelling and weather forecasting

Creative Applications

• Image generation (DALL-E, Stable Diffusion, Midjourney)
• Style transfer and image super-resolution
• Video generation and editing

---

When to Use Deep Learning

Deep learning is a powerful tool, but it is not always the right choice. Consider deep learning when:

• You have large amounts of data (thousands to millions of examples)
• The data is unstructured — images, text, audio, video
• You need to learn complex patterns that are hard to define manually
• You have access to GPU/TPU compute for training

Avoid deep learning when:

• You have a small dataset (hundreds of examples) — classical ML often works better
• The data is tabular with well-defined features — gradient boosting (XGBoost, LightGBM) typically outperforms deep learning
• You need a highly interpretable model (e.g., for regulatory compliance)
• Training time and compute cost are a major constraint

---

Summary

Deep learning is a branch of machine learning that uses multi-layered neural networks to automatically learn hierarchical representations from raw data. It excels at tasks involving images, text, audio, and other unstructured data. Key architectures include feedforward networks (MLPs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformers. The modern deep learning ecosystem — led by PyTorch and TensorFlow — provides powerful tools for building, training, and deploying models. While deep learning requires large datasets and significant compute, its ability to learn complex patterns has made it the foundation of modern AI.