Pick the most accurate one sentence summary of the tokenizer's role in fine-tuning.
A tokenizer is a fixed text to integer mapping whose IDs index the embedding matrix. The same tokenizer used at pretraining must be reused at fine-tuning and inference, or the embedding lookup pulls the wrong rows.
Picture a library where every book has a shelf number. The tokenizer is the librarian who tells you which shelf a request goes to. The shelves themselves are the model's giant lookup table of word meanings. If a new librarian shows up using a different numbering system but the shelves stay the same, asking for the cookbook now retrieves a math textbook. The shelves did not move; the numbers point somewhere else. That is what happens when you swap the tokenizer between training and serving. The text looks the same, but the integer IDs change. The lookup retrieves the wrong rows of meanings, and nothing the model says afterward makes sense.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The tokenizer is one of the most misunderstood components in a transformer pipeline. People often picture it as part of the model, perhaps a small neural module that learns alongside the transformer. It is none of that. The tokenizer is a deterministic algorithm with a frozen vocabulary file, produced once during pretraining and never updated again by gradient descent.
Understanding what the tokenizer is, and what it is not, prevents a class of fine-tuning bugs that are both common and hard to debug. The most common symptom is a model that produces garbled output starting from step zero. The root cause is almost always a tokenizer mismatch between the loaded weights and the loaded tokenizer.
What a tokenizer actually is
A tokenizer has two parts: an algorithm and a vocabulary. The algorithm is one of a handful of standard methods.
Byte-Pair Encoding (BPE) starts with single characters as tokens. It then repeatedly merges the most frequent adjacent pair into a new token, building up a vocabulary of common subword units. GPT models, Llama and Mistral use it.
WordPiece is similar to BPE but uses a likelihood-based criterion for selecting merges. The BERT family uses it.
SentencePiece Unigram treats tokenisation as a probabilistic model selection problem, starting with a large vocabulary and pruning unlikely tokens. Many multilingual models use it, including T5 and several modern open models.
In all three cases, the algorithm runs once on the pretraining corpus and produces a frozen vocabulary file. That file lists every subword token (typically 32,000 to 256,000 of them in modern LLMs) and assigns each one a fixed integer ID. The mapping never changes after pretraining.
At inference or fine-tuning time, the tokenizer applies its algorithm to incoming text and emits the corresponding sequence of integer IDs. There is no neural network involved, no gradient, no learning. It is a deterministic lookup plus a small amount of greedy or Viterbi subword segmentation logic.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3.1 ships with a tiktoken-based BPE tokenizer of 128,256 tokens. Fine-tunes that swap it for a different vocabulary produce immediately broken outputs.
- Loading another family's tokenizer at serving time (for example a Mistral tokenizer for a DeepSeek model) corrupts every input, not only rare scripts. Almost every ID points at the wrong embedding row.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through what happens when you call tokenizer.add_tokens followed by model.resize_token_embeddings, and why both calls are required.
add_tokens extends the tokenizer's vocabulary. Then resize_token_embeddings grows the embedding matrix and (usually) the lm_head so the new IDs have corresponding rows. Without both, encoding produces out of bounds IDs that crash the embedding lookup.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the tokenizer as a trainable component or as an optional preprocessing detail. It is a fixed mapping that has to match end to end. Forgetting that is one of the most common causes of broken fine-tunes.
60 second bullets to scan on the way to the call.
Definition of a tokenizer as a fixed text to integer mapping
Common tokenizer algorithms: BPE, WordPiece, SentencePiece Unigram
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.