Tokens, Costs & Rate Limits

Understanding tokens is fundamental to building cost-effective AI applications. This lesson explains how tokenisation works, how to count and estimate costs, and how to work within rate limits.

---

What Are Tokens?

Tokens are the fundamental units that language models process. A token is not the same as a word — it is a subword unit determined by the model's tokeniser.

Examples of Tokenisation

Text	Approximate Tokens
"Hello"	1 token
"Hello, world!"	3 tokens
"Tokenisation"	2–3 tokens
"supercalifragilistic"	5–6 tokens

Rules of thumb:

• ~4 characters per token in English
• ~¾ of a word per token
• Code tends to use more tokens per character than natural language
• Non-English languages often require more tokens per word

---

How Tokenisers Work

Modern LLMs use Byte Pair Encoding (BPE) or similar subword tokenisation:

1. Start with individual characters
2. Iteratively merge the most frequent pairs
3. Build a vocabulary of common subword units

"unbelievable" → ["un", "believ", "able"]     (3 tokens)
"AI"           → ["AI"]                         (1 token)
"GPT-4"        → ["G", "PT", "-", "4"]          (4 tokens)