Ask a chatbot how many times the letter “r” appears in “strawberry” and there’s a decent chance it will stumble. The people who built it aren’t fools,
and the model isn’t stupid. The trouble starts earlier, at a stage most users never see. Before a language model reads a single word, a small piece of software chops your text into fragments and swaps each fragment for a number. That software is the LLM tokenizer, and it quietly shapes almost everything about how a model behaves: what it costs, what it does well, and where it falls flat.
This guide walks through the whole mechanism in plain English. We’ll look at why models can’t simply read letters, how sub word tokenization algorithms like BPE, Word Piece and Unigram actually work, and what it all means when you’re paying per token or wondering why the same sentence costs more in Urdu than in English. No math’s degree required. If you’ve ever wondered what happens between your keyboard and the model’s “brain,” this is the missing chapter.
Table of Contents
- What Tokenization Actually Is
- Why Not Just Use Words or Letters?
- Subword Tokenization: The Middle Path
- Byte Pair Encoding (BPE) Explained With a Worked Example
- Byte-Level BPE: How GPT Models Handle Any Text
- WordPiece: The Approach Behind BERT
- Unigram and SentencePiece
- From Tokens to Numbers: Vocabulary, IDs and Embeddings
- Pre-Tokenization, Whitespace and Special Tokens
- Real-World Quirks: Strawberries, Numbers and Glitch Tokens
- The Language Fairness Problem
- Practical Tips for Writers, Developers and Teams
- Where Tokenization Is Heading
- Final Thoughts
- FAQ
What Tokenization Actually Is
Tokenization is the process of converting raw text into a sequence of integers that a neural network can process. That’s the whole definition. The details are where it gets interesting.
A model like GPT, Claude or Llama doesn’t work with text. It works with numbers. Under the hood, a neural network is an enormous pile of multiplication and addition, and you can’t multiply the word “banana” by a matrix. Something has to translate, and that translator is the LLM tokenizer.
The flow looks like this. Your text goes in as a string. The tokenizer splits it into chunks called tokens. Each token is looked up in a fixed vocabulary and replaced by its ID number. Those IDs go into the model. When the model replies, it produces IDs one at a time, and the process runs backwards to turn them into readable text. Every LLM tokenizer works in both directions, encoding on the way in and decoding on the way out.
Two facts are worth holding onto. First, the tokenizer is separate from the model. It’s built beforehand, frozen, and shipped alongside the model, and the two are a matched pair. Feed a model IDs from the wrong tokenizer and you get gibberish, like reading a book with the wrong alphabet. Second, the vocabulary is fixed. A model with a 100,000-token vocabulary can only ever read and write those 100,000 pieces, so everything else has to be assembled from combinations.
Here’s a toy illustration. The sentence “Tokenization is fun” might be split into pieces like “Token”, “inaction”, ” is” and ” fun”, each mapped to an ID such as 4421, 2065, 374 and 2523. Those numbers are made up for demonstration, and every real vocabulary assigns its own. What matters is the pattern: text becomes pieces, pieces become numbers. A well-designed LLM tokenizer is trained once on a big text sample and then behaves like a strict, unchanging dictionary.
Why Not Just Use Words or Letters?
If you were designing this from scratch, the two obvious options would be splitting by word or splitting by character. Both were tried, and both have serious problems. Understanding why they fail explains most design choices in a modern LLM tokenizer.
Word-level tokenization feels natural, since we think in words. But language is messy. English alone has hundreds of thousands of word forms once you count plurals, tenses, names, slang and technical terms. Add typos, URLs, code, hashtags and other languages, and the vocabulary balloons. Worse, any word missing from the vocabulary becomes an “unknown” token, and the model has no idea what it was. LLM tokenizer Earlier systems were full of these <UNK> placeholders, and they hurt translation quality badly, especially for languages with long compound words like German or heavily inflected ones like Turkish.
Character-level tokenization solves the unknown-word problem neatly. A few hundred characters cover almost everything, so nothing is ever out of vocabulary. The catch is length. A 500-word paragraph is roughly 3,000 characters, which means the model must process a sequence six times longer, and the cost of attention in a transformer grows roughly with the square of sequence length. Individual letters also carry almost no meaning, so the model burns capacity rebuilding words before it can think about ideas.
| Approach | Vocabulary Size | Sequence Length | Unknown Words | Main Weakness |
|---|---|---|---|---|
| Word-level | Huge (100k+) | Short | Common | Rare words break it |
| Character-level | Tiny (hundreds) | Very long | None | Slow, weak per-token meaning |
| Subword | Medium (30k–200k) | Moderate | Rare or none | Odd splits on some words |
The last row is where the field landed, and it’s what every current LLM tokenizer does in some form.
Subword Tokenization: The Middle Path
The core idea of subword tokenization is simple. Keep common words as single tokens, and break rare words into smaller, reusable pieces.
Take “unbelievably.” A subword system might split it into “un,” “believ” and “ably.” The model has probably never seen that exact word in some corners of its training data, but it has seen all three pieces many times, so it can infer a lot about the meaning. Now take a common word like “the.” It stays whole, because it appears constantly and deserves its own slot.
This gives you efficiency where text is predictable and flexibility where it isn’t. Names, misspellings, product codes and new slang can all be expressed, just less compactly. Nothing falls off the edge of the vocabulary.
The obvious next question is how the system decides which pieces deserve a slot. That’s where the algorithms come in. Three families dominate: Byte Pair Encoding, WordPiece and Unigram. They share the same goal and differ in how they learn the vocabulary and how they split new text. Let’s start with the most influential one.
Byte Pair Encoding (BPE) Explained With a Worked Example
BPE began life as a data compression trick, described by Philip Gage in 1994. In 2015, Rico Sennrich, Barry Haddow and Alexandra Birch adapted it for neural machine translation (the paper appeared at ACL 2016), and it has been the workhorse of language modelling ever since.
The training recipe is almost embarrassingly simple:
- Start with a vocabulary of individual characters.
- Count every pair of adjacent symbols across the training text.
- Merge the most frequent pair into a new single symbol.
- Repeat until the vocabulary reaches your target size.
Let’s run it by hand on a tiny corpus, using the classic example from the original paper. Imagine these words with these frequencies: “low” (5 times), “lower” (2), “newest” (6) and “widest” (3).
We begin by splitting everything into characters. “newest” is n-e-w-e-s-t, and so on. Now we count pairs. The pair “e” followed by “s” appears 9 times (6 in “newest,” 3 in “widest”), and “s” followed by “t” also appears 9 times. It’s a tie, so we pick one, say “es,” and merge it into a new symbol. Now the words read n-e-w-es-t and w-i-d-es-t.
Next round, “es” followed by “t” is the most common pair at 9, so we merge that into “est.” Then “l” followed by “o” (7 occurrences) becomes “lo,” and “lo” followed by “w” becomes “low.” After a few more rounds we’d also have “ne,” “new” and finally “newest” as a single token.
The result is an ordered list of merge rules: es, est, lo, low, ne, new, newest. That ordered list is the tokenizer. To encode a new word, you split it into characters and apply the merges in the order they were learned. Try “lowest,” a word that wasn’t in our training corpus. Start with l-o-w-e-s-t. Apply “es” to get l-o-w-es-t. Apply “est” to get l-o-w-est. Apply “lo” to get lo-w-est, then “low” to get low-est. The final tokens are “low” and “est.”
Notice what happened. The system never saw “lowest,” yet it handled it gracefully by reusing pieces it had learned from other words. That’s the magic of subword tokenization in a nutshell. Frequent patterns become single tokens, and everything else gets composed from parts.
A realistic training run does exactly this over billions of words for tens of thousands of merges. The early merges capture common letter pairs like “th” and “er.” The middle merges build syllables and short words. The late merges produce whole words, common word endings and even frequent phrases. If you ever inspect a trained vocabulary, you can practically watch the language being assembled layer by layer.
Byte-Level BPE: How GPT Models Handle Any Text
Classic BPE starts from characters, and that leaves a hole. Unicode has well over 100,000 characters, and a vocabulary that includes all of them wastes space on symbols that barely appear. Characters missing from training would still become unknowns.
Open AI’s GPT-2, released in 2019, popular isled a cleaner fix: run BPE on bytes instead of characters. Every piece of text, in any language or with any emoji, can be written as UTF-8 bytes, and there are only 256 possible byte values. Start with those 256 as your base alphabet, and every possible string can be represented. Nothing is ever unknown. That property is the reason byte-level BPE became the default for so many modern systems, and it’s why the modern LLM tokenizer almost never chokes on unusual input.
GPT-2’s vocabulary had 50,257 entries: 256 base bytes, 50,000 learned merges and one special end-of-text token. Later OpenAI encodings grew. The cl100k_base encoding used by GPT-3.5 and GPT-4 has roughly 100,000 tokens, and o200k_base, used by GPT-4o, has roughly 200,000. Meta’s Llama 3 moved to a 128,000-token vocabulary as well, after Llama 2 used 32,000. Bigger vocabularies are a clear industry trend, and we’ll see why in a moment.
The byte-level approach has a side effect you can observe directly. Common characters like plain English letters take one byte and usually merge into larger tokens. A rare character, or an emoji, may take three or four bytes and might be split across several tokens. So a single smiley can genuinely cost you more than one token. The same LLM tokenizer that handles English elegantly may spend several tokens on a single character of a less common script.
WordPiece: The Approach Behind BERT
WordPiece was developed at Google and became famous through BERT, whose vocabulary has about 30,000 entries. It looks similar to BPE from the outside, but the details differ in two useful ways.
First, the notation. WordPiece marks pieces that continue a word with a double hash prefix. The word “playing” becomes “play” plus “##ing.” The “##” tells you this piece attaches to whatever came before it. You’ll see it constantly if you ever print BERT’s output, and it’s an easy way to recognise a WordPiece system on sight.
Second, the training criterion. BPE always merges the most frequent pair. WordPiece instead scores each candidate pair by how much merging it improves the likelihood of the training data, which works out roughly to the pair’s frequency divided by the product of its parts’ individual frequencies. In practice that means it favours pairs whose pieces belong together more than chance would suggest, rather than pairs that are merely common because their components are common. It’s a subtle difference, but it changes which pieces make it into the vocabulary.
At encoding time, WordPiece uses a greedy longest-match-first strategy. It looks for the longest vocabulary entry that matches the start of the word, takes it, then repeats on what’s left. If nothing matches at some point, the whole word collapses to an unknown token, which is one reason byte-level systems are more forgiving. Anyone comparing a WordPiece-style LLM tokenizer with a byte-level one should keep this behaviour in mind, especially when handling rare characters or noisy text.
Unigram and SentencePiece
Unigram takes the opposite journey. Proposed by Taku Kudo in 2018, it starts with a large pool of candidate pieces and prunes downward instead of building upward.
Here’s the gist. Begin with an oversized vocabulary, perhaps every frequent substring in the data. Assign each piece a probability. Then repeatedly ask which pieces could be removed with the smallest damage to the overall likelihood of the training corpus, drop the least useful ones, and re-estimate. Continue until the vocabulary shrinks to the target size.
One distinctive feature of Unigram is that a single word can usually be segmented in several valid ways. The tokenizer picks the most probable segmentation, often using the Viterbi algorithm, but it can also sample alternatives during training. That technique, called subword regularisation, exposes the model to slightly different splits of the same text and can make it more robust. It’s a genuinely different philosophy from BPE’s single deterministic merge order.
You’ll often see Unigram mentioned alongside SentencePiece, and the two get confused. SentencePiece, released by Kudo and John Richardson in 2018, is a software library, not an algorithm. It implements both BPE and Unigram. Its big contribution is treating input as a raw stream of characters with no assumed word boundaries. Whitespace is simply encoded as a visible symbol, typically “▁,” so decoding can restore the original text perfectly. That makes it well suited to languages such as Japanese and Chinese, which don’t put spaces between words. T5 and ALBERT use Unigram through SentencePiece, and the original Llama models used SentencePiece with BPE.
If you’re comparing options, the practical takeaway is that the choice of algorithm matters less than the training data, the vocabulary size and the pre-processing rules. A thoughtfully trained LLM tokenizer of any of these types can perform well, and a sloppy one will hurt whichever family it belongs to.
| Algorithm | Direction | Used By | Notable Trait |
|---|---|---|---|
| BPE / byte-level BPE | Bottom-up merging | GPT models, Llama 3 | No unknown tokens at byte level |
| WordPiece | Bottom-up, likelihood-based | BERT and relatives | “##” continuation markers |
| Unigram (via SentencePiece) | Top-down pruning | T5, ALBERT | Multiple valid splits |
From Tokens to Numbers: Vocabulary, IDs and Embeddings
So far we’ve treated the token ID as the end of the story. It’s really the start of the model’s story.
A token ID is just an index, a row number. The first layer of the network is an embedding table, a giant matrix with one row per vocabulary entry and one column per dimension of the model’s internal representation. When token 2065 arrives, the model fetches row 2065, a list of, say, 4,096 learned numbers, and that vector is what the rest of the network actually works with. During training, those vectors get adjusted so that tokens used in similar ways end up with similar coordinates.
This is why vocabulary size is a real engineering trade-off. Take a 100,000-token vocabulary and a 4,096-dimension embedding: that’s about 410 million parameters in the table alone, and there’s usually a matching output layer that scores every token at each step. A bigger vocabulary makes text more compact, so the same sentence takes fewer tokens, which means faster generation, lower cost and more effective context. But it also enlarges those tables, and rare tokens get seen so seldom in training that their vectors stay poorly learned. A smaller vocabulary does the reverse. Every team choosing an LLM tokenizer is balancing those forces, which is why numbers have crept from 32,000 up to 128,000 and 200,000 as hardware and data have improved.
Another point that surprises people: the model has no built-in knowledge of what a token looks like. It doesn’t know that “cat” is spelled c-a-t. It learns associations between token vectors through exposure to text, and any spelling knowledge is inferred indirectly. Keep that thought, because it explains the strawberry problem coming up shortly.
Pre-Tokenization, Whitespace and Special Tokens
Before the merge rules ever run, most systems apply a pre-tokenization step. This is a set of rules, often a regular expression, that cuts the text into rough chunks so merges can’t cross certain boundaries. GPT-2’s pattern, for example, keeps letters, digits and punctuation from merging into one another. Without it, you might see a token that glues the end of a word to a comma and the start of the next word, which would waste vocabulary on accidental combinations.
Whitespace handling deserves a moment because it trips people up. In most GPT-style vocabularies, the space before a word is attached to the word. So ” hello” (with the leading space) and “hello” (without) are two different tokens with two different IDs. Capital letters count too: “Hello,” ” hello” and “HELLO” are all separate entries or split differently. This is why tiny formatting changes in a prompt can subtly change model behaviour. The model isn’t seeing the same input you think it is.
Then there are special tokens. These are reserved vocabulary entries that mark structure rather than represent text: an end-of-text marker such as <|endoftext|>, tokens signalling the start or end of a sequence, and in chat models, markers that separate system, user and assistant messages. When you send a chat message, the app wraps it in these markers behind the scenes, and they consume tokens too. Every well-designed LLM tokenizer reserves a handful of these, and they’re a big reason a “short” conversation can use more of your budget than the visible words suggest.
Real-World Quirks: Strawberries, Numbers and Glitch Tokens
Once you understand the mechanics, several famous model failures stop being mysterious.
The strawberry problem. When you ask about the letters in “strawberry,” the model doesn’t see ten letters. Depending on the vocabulary, it sees a few chunks, something like “str,” “aw” and “berry.” It was never handed the individual characters, so counting them requires knowledge it must have picked up indirectly. The tokenizer hides the very thing you’re asking about. It’s the text equivalent of asking someone to count the bricks in a wall they can only see through frosted glass. Newer models handle this better, partly through training and partly through reasoning steps that spell words out, but the underlying cause is a tokenization boundary. This is the clearest everyday example of how an LLM tokenizer shapes behaviour.
Numbers. Early tokenizers chopped numbers up inconsistently. A number like 1234567 might split as “123,” “456,” “7,” while 1234568 splits differently, which makes arithmetic patterns hard to learn. Some newer models, Llama among them, split numbers into individual digits, which gives the model a more consistent structure to work with. It’s a nice example of a design decision in tokenization directly affecting maths ability.
Glitch tokens. In 2023, researchers studying GPT models found strange vocabulary entries such as ” SolidGoldMagikarp,” apparently derived from a Reddit username. The token existed because the text used to build the vocabulary contained that string often, but the model’s own training data barely did. The token’s vector never got properly trained, and asking about it produced bizarre, evasive or nonsensical replies. It’s a reminder that the tokenizer and the model are trained separately and sometimes on different data, and mismatches leave fingerprints.
Code and indentation. Programming languages are full of repeated whitespace. If a tokenizer treats every space as a separate token, indented code becomes expensive fast. Newer vocabularies include tokens for runs of spaces and common code patterns, which is part of why recent models handle programming text so much more economically.
The Language Fairness Problem
Here’s the part that deserves more attention than it gets. Most tokenizers are trained on data dominated by English. English words are therefore well represented, and text in other languages gets fragmented into far more pieces.
The practical consequences are real. The same sentence in Hindi, Arabic, Urdu or Thai can take noticeably more tokens than its English equivalent, often two to several times more depending on the tokenizer. Since usage is priced per token, speakers of those languages pay more for the same information. Since context windows are measured in tokens, they also fit less content. And since generation happens token by token, responses can be slower too.
There’s a quality angle as well. When a word is shattered into many tiny fragments, the model has to piece meaning together from weaker signals. Research on multilingual models has repeatedly linked heavier fragmentation to weaker performance in those languages.
The picture is improving. Larger vocabularies and more balanced training data have narrowed the gap. GPT-4o’s move to roughly 200,000 tokens, for example, was partly aimed at handling non-English text more efficiently. But if you’re building a product for a multilingual audience, testing your LLM tokenizer on real samples of your users’ languages before committing is one of the smartest checks you can do.
Practical Tips for Writers, Developers and Teams
You don’t need to train a tokenizer to benefit from understanding one. Here’s what to do with all of this.
Use the rule of thumb, then verify. For English, one token is roughly four characters, or about three quarters of a word. So 100 tokens is around 75 words. That’s fine for back-of-envelope budgeting, but code, numbers and non-English text can deviate wildly, so measure when it matters.
Inspect real tokenization. With OpenAI’s open-source tiktoken library, it takes a few l
The second line prints each token as text, so you can see exactly where the splits fall. Hugging Face’s tokenizers and transformers libraries offer the same for open models. Running your own prompts through an LLM tokenizer like this is the fastest way to build intuition.
Trim the waste. Boilerplate, repeated instructions, bloated JSON with long key names and unnecessary whitespace all cost tokens. Shortening field names or cleaning formatting can cut costs without changing meaning.
Mind your spaces and casing. Because ” word” and “word” differ, a stray trailing space at the end of a prompt can nudge a model toward odd completions. If you’re seeing strange behaviour, check the boundaries.
Never mix tokenizers and models. Each model expects its own vocabulary. If you fine-tune or self-host, use the tokenizer that shipped with the model, and treat any change to it as a retraining decision, not a settings tweak.
Budget in tokens, not words. Context limits, latency and price are all token-denominated. Teams that plan around words tend to get surprised, especially with multilingual content.
Where Tokenization Is Heading
Tokenization has always been a slightly awkward stage. It’s a hand-built, statistical preprocessing step bolted onto an otherwise learned system, and it causes the quirks above. So researchers keep asking whether it can be removed.
Byte-level models such as Google’s ByT5 feed raw bytes straight into the network, avoiding vocabularies entirely at the cost of much longer sequences. More recently, Meta researchers proposed the Byte Latent Transformer, which groups bytes into variable-sized patches based on how predictable the next byte is, spending more compute where text is hard and less where it’s easy. Results are promising, but subword systems still power nearly every major model in production today.
For the foreseeable future, then, the LLM tokenizer remains a core part of the stack. Even if tokenizer-free approaches mature, the questions they answer, how to balance sequence length, vocabulary and fairness, won’t disappear. They’ll just move somewhere else.
Final Thoughts
Here’s the idea worth carrying away. A language model never reads your words. It reads numbers produced by a tokenizer that decided, long before your prompt existed, which fragments of language deserved a place in the vocabulary. That decision explains why some prompts cost more than others, why maths and spelling can go strangely wrong, and why the same sentence can be cheap in one language and pricey in another.
Once you know how the LLM tokenizer works, model behaviour that once looked random starts to make sense. So try it yourself. Paste a few of your real prompts into a tokenizer viewer, look at where the splits land, and compare English against another language you care about. Ten minutes of that will teach you more than any diagram. Every time you use a model from now on, you’ll know there’s a LLM tokenizer quietly working between you and the answer.
FAQ
What is an LLM tokenizer?
An LLM tokenizer is the component that converts text into a sequence of numeric IDs a language model can process, and converts the model’s output IDs back into text. It uses a fixed vocabulary of pieces, usually whole words, word fragments or bytes, learned from a large text sample before the model is trained.
What is subword tokenization and why is it used?
Subword tokenization splits text into pieces that sit between whole words and single characters. Common words stay intact, while rare words are broken into reusable parts. This keeps the vocabulary manageable, keeps sequences short and avoids unknown words, which is why almost every modern LLM tokenizer relies on it.
What is the difference between BPE, WordPiece and Unigram?
BPE builds its vocabulary by repeatedly merging the most frequent pair of symbols. WordPiece also builds upward but picks merges that best improve the likelihood of the training data. Unigram starts with a large vocabulary and prunes it down, and it can produce several valid splits for the same word. All three are subword algorithms, and each LLM tokenizer built on them differs mainly in training method and encoding behaviour.
How many characters or words is one token?
For typical English text, one token is about four characters or roughly three quarters of a word, so 1,000 tokens is around 750 words. The ratio changes with the language, the tokenizer and the content. Code, numbers and non-English text often use more tokens per word, so it’s worth testing your own text in the specific LLM tokenizer you plan to use.
Why do language models struggle with counting letters?
Models see tokens, not individual letters. A word like “strawberry” arrives as a few chunks, so the model has no direct view of its spelling. It has to infer letter-level facts from patterns learned in training, which is unreliable. That’s a tokenization limit rather than a lack of intelligence.
Does tokenization affect the price of using an AI model?
Yes. Most providers charge per token for both input and output, and context limits are also counted in tokens. A text that splits into more tokens costs more and fills the context window faster, so an efficient LLM tokenizer directly lowers cost and improves how much you can fit in a prompt.
Read About 7 AI Tools for Small Business Marketing:That Save Time and Grow Sales