Neural Networks & Deep Learning

Neural networks are a class of machine learning models inspired by the structure of the biological brain. Deep learning refers to neural networks with many layers, enabling them to learn complex, hierarchical representations of data.

---

Biological Inspiration

The human brain contains approximately 86 billion neurons, each connected to thousands of others via synapses. Artificial neural networks borrow this concept in a simplified form:

Biological	Artificial
Neuron	Node (unit)
Synapse	Connection (weight)
Electrical signal	Numerical value
Signal strength	Weight value
Firing threshold	Activation function

Note: While inspired by biology, artificial neural networks are a significant simplification. Modern research focuses on what works mathematically rather than strict biological accuracy.

---

The Artificial Neuron (Perceptron)

The simplest neural network unit is the perceptron, introduced by Frank Rosenblatt in 1958.

Artificial Neuron:

  x1 ──w1──┐
            │
  x2 ──w2──┼──▶ [ Sum(weighted inputs) + bias ] ──▶ [ Activation f() ] ──▶ output
            │
  x3 ──w3──┘

  Inputs    Weights        Summation                  Activation         Output

Each neuron receives inputs, multiplies each by a weight, sums the results with a bias, and passes the result through an activation function.

Activation Functions

Activation	Range	Common Use
Sigmoid	(0, 1)	Binary classification output
Tanh	(-1, 1)	Hidden layers (centred output)
ReLU	[0, infinity)	Most popular for hidden layers
Leaky ReLU	(-infinity, infinity)	Avoids "dying ReLU" problem
Softmax	(0, 1), sums to 1	Multi-class classification output

Tip: ReLU (Rectified Linear Unit) is the default choice for hidden layers. It is computationally efficient and avoids the vanishing gradient problem.

---

Network Architecture: Layers

    Neural Network Architecture
    ===========================

    Input        Hidden         Hidden        Output
    Layer        Layer 1        Layer 2       Layer

    O───────────O              O
     \         / \            / \
      \       /   \          /   \
    O──+─────O     O────────O     O─────────O
      /       \   /          \   /
     /         \ /            \ /
    O───────────O              O
      \         / \            /
       \       /   \          /
    O──+─────O     O────────O

   (features)   (learned        (learned      (prediction)
                 representations)

• Input layer: Receives the raw features (one node per feature)
• Hidden layers: Each learns increasingly abstract representations
• Output layer: Produces the final prediction

A deep neural network has two or more hidden layers — that is what gives deep learning its name.

---

Training: Backpropagation and Gradient Descent

Forward Pass

Data flows forward through the network, producing a prediction compared to the true label via a loss function.