Neural Networks & Deep Learning

Neural networks are a class of machine learning models inspired by the structure of the biological brain. Deep learning refers to neural networks with many layers, enabling them to learn complex, hierarchical representations of data.

---

Biological Inspiration

The human brain contains approximately 86 billion neurons, each connected to thousands of others via synapses. Artificial neural networks borrow this concept in a simplified form:

Biological	Artificial
Neuron	Node (unit)
Synapse	Connection (weight)
Electrical signal	Numerical value
Signal strength	Weight value
Firing threshold	Activation function

Note: While inspired by biology, artificial neural networks are a significant simplification. Modern research focuses on what works mathematically rather than strict biological accuracy.

---

The Artificial Neuron (Perceptron)

The simplest neural network unit is the perceptron, introduced by Frank Rosenblatt in 1958.

graph LR
  X1["x1"] -- w1 --> S["Sum(weighted inputs) + bias"]
  X2["x2"] -- w2 --> S
  X3["x3"] -- w3 --> S
  S --> F["Activation f()"]
  F --> O["output"]

Each neuron receives inputs, multiplies each by a weight, sums the results with a bias, and passes the result through an activation function.

Activation Functions

Activation	Range	Common Use
Sigmoid	(0, 1)	Binary classification output
Tanh	(-1, 1)	Hidden layers (centred output)
ReLU	[0, infinity)	Most popular for hidden layers
Leaky ReLU	(-infinity, infinity)	Avoids "dying ReLU" problem
Softmax	(0, 1), sums to 1	Multi-class classification output

Tip: ReLU (Rectified Linear Unit) is the default choice for hidden layers. It is computationally efficient and avoids the vanishing gradient problem.

---

Network Architecture: Layers

graph LR
  subgraph Input Layer
    I1["O"]
    I2["O"]
    I3["O"]
  end
  subgraph Hidden Layer 1
    H1["O"]
    H2["O"]
  end
  subgraph Hidden Layer 2
    G1["O"]
    G2["O"]
  end
  subgraph Output Layer
    P1["O"]
  end
  I1 --> H1
  I1 --> H2
  I2 --> H1
  I2 --> H2
  I3 --> H1
  I3 --> H2
  H1 --> G1
  H1 --> G2
  H2 --> G1
  H2 --> G2
  G1 --> P1
  G2 --> P1

• Input layer: Receives the raw features (one node per feature)
• Hidden layers: Each learns increasingly abstract representations
• Output layer: Produces the final prediction