Definition
Tokenization splits text into the small units a model actually reads, usually word fragments rather than whole words. An embedding is what those units become afterwards; tokenization is the cut that happens first. Every limit and cost in an AI system is counted in tokens, not in words.
At a glance
- CategoriesTechnical
- Related fieldsAI Search, Technical
- DifficultyIntermediate
At a glance
- A token is a fragment of text: common words survive whole, unusual ones get split.
- Every model has its own tokenizer, and the same sentence divides differently between them.
- Token counts drive context limits and pricing, which is why nothing in AI is billed per word.
Where the cuts fall
A tokenizer is built by scanning a large body of text and keeping the fragments that turn up most often. What appears constantly becomes one token. What is rare is assembled from smaller pieces.
The result is lopsided. Ordinary English words come through intact. A technical term, a surname or a product code breaks into fragments, and the model sees those rather than the word.
The main approaches are set out in the Hugging Face tokenizer summary. All of them split before any meaning is computed: the fragments are what the model reads, and an embedding is built from them.
A rare brand name splits badly
Most of this never touches you. The exception is your own name, and names are what gets split.
A brand like Saucony or Wüsthof is rare in training text, so it arrives as fragments rather than a unit. Two things get likelier: the model misspells it when writing from memory, and the step deciding which real thing the name refers to has a weaker signal, which is what named entity recognition does.
You see it as an assistant producing a near-miss version of your name, or attaching it to the wrong company.
Honest scale: a small effect underneath much larger ones. Fragmented names get remembered fine when they appear often enough in consistent context. Volume beats spelling.
Keep your names in one piece
Three habits, all cheap.
- Write the full name in plain text on every important page. Not only in the logo or an alt attribute. A name that never appears as text never gets tokenised.
- Keep spelling and spacing identical everywhere. “ProSeries”, “Pro Series” and “Pro-Series” split differently and read as three things.
- Give the name context on first use. Pair it with your category in the same sentence, so the fragments arrive attached to something a model can place.
Frequently asked questions
-
Is a token the same as a word?
No. A token is a fragment and the relationship to words is uneven. Common short words are one token each. A long or unusual word becomes several. Punctuation and spaces count too, so token counts and word counts never line up.
-
Do I need to count tokens for anything?
Only if you are building with an API, where tokens set cost and input limits. For marketing work there is nothing to count. Anyone selling token optimization for content is selling a number with no bearing on whether you get cited.
-
Does tokenization change whether my content gets cited?
Barely, and it is the wrong thing to spend attention on. Citation is decided by retrieval and by whether your passage answers the question. Tokenization sits well below that.
-
Why does an assistant misspell my brand name?
Usually because the name is rare in training text and arrives as fragments, so the model reassembles it from what it half-remembers. It happens most when no search ran, and least when your name appears consistently across many sources, which a large language model learns from directly.