A tokenizer turns text into a list of integer **token IDs**, and the model learns one embedding vector for each ID during pretraining. The input embedding table has shape `[vocab_size, d_model]`, which means one row per token ID.
The problem with swapping the tokenizer is that the IDs are not just labels. They are the addresses the model uses to look up learned meaning. If a new tokenizer assigns the word "hello" to a different ID, the model still fetches the embedding row at the old address, which now belongs to a different token entirely.
There is a second component with the same dependency. The output head, called `lm_head`, has shape `[d_model, vocab_size]` and turns the model's final hidden state into a score for every possible next token. A bigger vocabulary means new rows here too, and those new rows have no trained weights.
So it is not a memory limit or a licensing rule. The learned weights are tied to a specific ID layout, and changing that layout breaks the lookup on both ends of the model. That is why the tokenizer is effectively chosen once, at the start of pretraining, and frozen after.
You cannot hot-swap a tokenizer because two model components are indexed by token ID, and changing the vocabulary scrambles that index.
The input embedding table has shape `[vocab_size, d_model]`, one learned row per ID. The output projection `lm_head` has the mirror shape `[d_model, vocab_size]`. Both were trained against one specific mapping from token to ID.
When a new tokenizer remaps IDs, the row that used to mean "the" now sits at a different position. So every embedding lookup returns the wrong learned vector. A larger vocabulary also adds brand-new IDs that have no trained rows at all, on both the input and output side.
The key insight is that token IDs are not arbitrary labels you can relabel. They are the literal addresses the model uses to fetch and produce meaning, and the addresses are baked into the weights. So the practical answer is that tokenizer choice is **sticky**: it is fixed at the start of pretraining. To change it you either retrain from scratch, or run a vocabulary-adaptation step that initializes the new rows and continues training, which is far cheaper but still not free.
Quick reference
TL;DR
A pretrained tokenizer is frozen because the embedding matrix and lm_head are indexed by token ID; changing the vocabulary remaps IDs and points every learned weight at the wrong token.
Imagine a library where every book has a numbered shelf, and the librarian has memorized exactly which book sits at each number. Now someone renumbers all the shelves overnight. The librarian still walks to the same numbers, but every shelf now holds a different book, so every request comes back wrong. The model's tokenizer is that numbering scheme. The learned embeddings are the librarian's memory of what lives at each number. Swap the tokenizer and you renumber the shelves, but the model keeps reaching for the old numbers, so it pulls the wrong meaning every single time.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Trying to fix multilingual fertility by dropping a better tokenizer into a shipped checkpoint The embedding matrix and lm_head are indexed by the original token IDs, so a swap points every learned row at the wrong token and produces garbage without retraining.
Relabeling existing token IDs to make room for new tokens in the middle of the range Any reordering of existing IDs misaligns the trained embedding rows; only appending new IDs at the end preserves the old mapping.
Assuming continued pretraining alone repairs a full tokenizer replacement cheaply Replacing rather than extending the vocabulary invalidates every existing row at once, so the continued-training bill approaches a fresh pretraining run rather than a light adaptation.
Comparing two checkpoints by reusing one tokenizer on the other's weights Each checkpoint's weights were trained against its own ID layout, so cross-applying a tokenizer makes the comparison meaningless and the outputs incoherent.
Quick scan · 2–4 min
Everything important, quickly.
3 min: vocab_size lives in two tensors + token IDs are addresses + remap breaks lookups + new rows untrained + expansion vs adaptation vs retrain + why it is sticky.
Real-world examples
Real products, models, and research that use this idea.
1OpenAI shipped o200k_base as a new tokenizer for GPT-5.5 rather than retrofitting cl100k_base into older checkpoints, because the embedding and head weights are tied to the original ID layout.
2Meta's Llama 4 line uses an expanded multilingual vocabulary chosen before pretraining, since the tokenizer cannot be changed once the Llama embedding matrix is trained.
3Teams adapting an English Llama checkpoint to a new language commonly use embedding initialization by averaging plus continued pretraining instead of swapping tokenizers outright.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QSuppose you only want to add 5,000 new tokens for a new domain without disturbing existing behavior. How do you do it?
Sign in or go Pro to see the approach for this follow-up.
QHow would you initialize the embeddings for brand-new tokens so continued training converges faster?
Sign in or go Pro to see the approach for this follow-up.
QWhy can't you just learn a linear map from old token IDs to new ones and reuse the embeddings?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking the blocker is memory or licensing, when the real issue is that token IDs remap and the learned embedding rows no longer line up with the tokens they were trained on.