Tokenization Strategies for LLMs: BPE, WordPiece, and Unigram Explained

You type a prompt into an AI chatbot, hit enter, and get an answer. But before that model can even think about your question, it has to chop your words into tiny pieces called tokens. This process, known as tokenization, is the unsung hero of modern AI. It determines how efficiently a model reads, how much it costs to run, and why some languages work better than others. If you’ve ever wondered why a simple sentence sometimes eats up half your context window, or why GPT-4 handles English differently than Turkish, the answer lies in the specific tokenization strategy used under the hood.

Why We Don’t Just Use Words

It seems obvious: humans read words, so why don’t computers? The problem is that language is messy. New slang pops up daily, typos happen, and domain-specific jargon changes constantly. If a model had a fixed vocabulary of every possible word, it would be enormous and slow. Worse, it wouldn’t know what to do with "un-believ-able" if it hadn’t seen that exact string before.

Enter subword tokenization. Instead of treating "unbelievable" as one unit, a tokenizer might break it into "un", "believe", and "able". This allows the model to understand the root meaning even if it’s never encountered the full word. This approach, pioneered in the late 2010s, transformed NLP from brittle rule-based systems to flexible neural networks. By breaking text into manageable units, we transform variable-length sentences into fixed-dimensional inputs that neural nets can actually process.

Byte-Pair Encoding (BPE): The Industry Standard

If you’re using ChatGPT, you’re likely interacting with a BPE (Byte-Pair Encoding) tokenizer. Developed originally for data compression in the 1990s by Philip Gage, BPE was adapted for NLP because it’s surprisingly effective at balancing two competing needs: keeping the vocabulary small enough to be fast, but large enough to cover most words without chopping them into meaningless characters.

Here’s how it works: Start with a base vocabulary of all individual characters (bytes). Then, look at your training data and find the most frequent pair of symbols. Merge them into a new single token. Repeat this process thousands of times until you reach your target vocabulary size-typically around 50,000 tokens for OpenAI’s models. Because it prioritizes frequency, BPE naturally learns common prefixes and suffixes. For example, it will quickly learn that "ing" appears after many verbs, creating a distinct token for it.

BPE is the dominant choice, used in roughly 68% of commercial LLMs. Why? It’s robust. It handles unseen words gracefully by falling back to smaller subwords or characters. However, it’s not perfect. BPE tends to have a lower "fertility score" (average tokens per word) compared to other methods, which is good for speed, but it can sometimes create arbitrary splits that confuse the model’s understanding of morphology.

WordPiece: The Likelihood-Based Alternative

While BPE looks at frequency, WordPiece chooses merges based on likelihood scores. Developed by Google for use in BERT, this method asks a different question: "Which merge maximizes the probability of the training corpus?" rather than just "Which pair appears most often?"

This subtle shift in logic leads to different results. WordPiece often produces a higher fertility score-around 1.7 tokens per word compared to BPE’s 1.4. What does that mean for you? Longer sequences. If you’re processing a paragraph, WordPiece might generate 18% more tokens than BPE for the same text. That sounds bad for cost, right? Not necessarily. Those extra tokens can preserve more granular linguistic information, which is beneficial for tasks requiring deep semantic analysis, like sentiment detection or named entity recognition.

However, this granularity comes at a price. In multilingual settings, WordPiece can struggle more than BPE because it doesn’t handle rare character combinations as flexibly. If you’re building a model primarily for English classification tasks, WordPiece remains a strong contender. But for general-purpose generation, where speed and token economy matter, it often loses out to BPE.

Unigram and SentencePiece: Compression Masters

If BPE is the reliable sedan and WordPiece the detailed sports car, Unigram is the fuel-efficient hybrid. Unlike BPE and WordPiece, which build vocabularies by merging upwards, Unigram starts with a huge candidate set and prunes downwards. It uses a probabilistic model to determine which subwords are most useful, keeping only those that maximize the likelihood of the training data.

Unigram is implemented in tools like SentencePiece, which is language-independent and operates directly on raw bytes. This makes it exceptionally good at handling diverse languages and special characters without needing pre-tokenization steps. Research shows Unigram achieves superior compression efficiency, with fertility scores averaging 1.2 tokens per word. That means fewer tokens for the same input, leading to faster inference and lower API costs.

Where does Unigram shine? Code. Machine code and programming languages have rigid structures and unique symbols. A study comparing tokenizers on machine code datasets found Unigram provided 22% better compression than BPE and 31% better than WordPiece. If you’re working with Python, C++, or assembly, a Unigram-based tokenizer like SentencePiece is often the smarter choice. It treats the text as a stream of bytes, avoiding the pitfalls of assuming space-separated words exist in code.

The Cost of Language: Why English Wins

Here’s a controversial truth: most LLMs are biased toward English. And tokenization is a big reason why. Because major models were trained predominantly on English data (92.7% of surveyed open-source models), their vocabularies are optimized for English morphemes. When you feed them German, Turkish, or Swahili, the tokenizer struggles to find efficient subword representations.

The result? Your non-English text gets chopped into way more pieces. While English might average 1.3 tokens per word, Turkish could require 2.1 tokens per word-a 61% discrepancy. This isn’t just an academic statistic; it hits your wallet. Dr. Limisiewicz’s research indicates that generating 100 characters in a non-primary language can cost up to 43% more in computational resources. Cisco’s enterprise data backs this up, showing a 37% increase in API costs for non-English requests due to these inefficiencies.

This bias creates a performance gap too. Models may understand English nuances deeply while skimming over low-resource languages because the signal is diluted across more tokens. Recent efforts, like Meta’s Llama 3.2 with its 128,000-token vocabulary, aim to fix this. They reduced the English-to-Swahili tokenization gap from 37% to 22%, but the disparity remains significant.

Choosing the Right Strategy for Your Project

So, which tokenizer should you pick? It depends on your goals.

• General Purpose Chatbots: Stick with BPE. It’s battle-tested, widely supported, and offers the best balance of speed and quality for mixed-language or English-heavy applications.
• Code Generation & Analysis: Go with Unigram/SentencePiece. Its byte-level processing handles syntax and symbols better, reducing sequence length and improving accuracy on structural tasks.
• Fine-Grained Classification: Consider WordPiece. If you need the model to catch subtle distinctions in phrasing and don’t mind slightly longer sequences, its likelihood-based merging can help.
• Multilingual Apps: Look for models with larger vocabularies (128k+ tokens) or dynamic allocation strategies. Avoid small-vocabulary tokenizers (like 32k) if you expect heavy non-English usage, as they will inflate your token counts dramatically.

Remember, you don’t always have to train from scratch. Starting with a pre-trained tokenizer and fine-tuning on your domain data reduces implementation time by 63% compared to building a new vocabulary from zero. Also, watch out for whitespace handling. Improper management of spaces can inflate token counts by 15-20%, silently eating your budget.