Sequence-to-Sequence Models

Sequence-to-sequence (Seq2Seq) models transform an input sequence into an output sequence, where the two sequences can have different lengths. They are the foundation of machine translation, text summarisation, question answering, and many other NLP tasks.

---

What is Sequence-to-Sequence?

Component	Role
Encoder	Reads the input sequence and compresses it into a fixed-size context vector
Decoder	Takes the context vector and generates the output sequence one token at a time

Applications

Task	Input	Output
Machine translation	"The cat sat on the mat"	"Le chat s'est assis sur le tapis"
Text summarisation	Long article	Short summary
Question answering	Context + question	Answer
Chatbots	User message	Bot response
Text-to-SQL	"Show all users"	"SELECT * FROM users"
Code generation	Natural language description	Code snippet

---

The Encoder-Decoder Architecture

Basic Seq2Seq with RNNs

The original Seq2Seq model (Sutskever et al., 2014) uses two RNNs: