Definition
Tokenization is how a model breaks text into small pieces called tokens, the units it actually reads and generates. A token is often a word fragment rather than a whole word.
On this page
At a glance
- Categories Technical
- Related fields AI Search, Technical
- Difficulty Intermediate
How Tokenization Works
Models do not read letters or whole words exactly. They split text into tokens, which might be a short word, part of a longer word, or a piece of punctuation. Everything a model does is counted in tokens, including the size of its context window and the cost of running it.
Why Tokenization Matters
You rarely need to think about tokens directly, but the concept explains a lot: why context windows are described in tokens, why concise writing is cheaper for engines to process, and why very long pages can strain retrieval.
Frequently Asked Questions
-
What exactly is a token?
A small chunk of text, often a word fragment. Roughly speaking, a token is about three quarters of a word in English.
-
Do I need to optimize for tokens?
No. Clear, concise writing naturally tokenizes efficiently. There is no token trick worth chasing.