Transformers and Attention

The Transformer architecture, introduced in the 2017 paper Attention Is All You Need by Vaswani et al., revolutionised NLP and machine learning. Transformers replaced recurrent networks with self-attention, enabling parallel processing, better long-range dependency modelling, and massive scalability.

---

Why Transformers?

RNNs and LSTMs process text sequentially — one token at a time. This creates two major problems:

Problem	Description
Slow training	Cannot parallelise across sequence positions
Long-range dependencies	Information from early tokens fades as the sequence grows

Transformers solve both problems using self-attention, which processes all tokens in parallel and directly connects every position to every other position.

---

The Transformer Architecture

The Transformer consists of an encoder stack and a decoder stack, each made of repeated layers.

Encoder Layer

Each encoder layer contains:

1. Multi-Head Self-Attention — allows each token to attend to all other tokens
2. Feed-Forward Network — two linear layers with a ReLU activation
3. Layer Normalisation and Residual Connections around each sub-layer

Decoder Layer