A team has a pretrained LLM checkpoint trained with a 50,000-token vocabulary. They want to swap to a new tokenizer with a 100,000-token vocabulary that has better multilingual coverage. Explain precisely what breaks at the model architecture and weight level, and describe what remediation options exist with their costs.
Free · 2 AI evals / day
A tokenizer assigns every text piece an integer **token ID**, and the model learns one embedding vector per ID. The input embedding table has shape `[vocab_size, d_model]`, so there is exactly one row for each ID.
When you swap to a new tokenizer, the IDs remap. The word "hello" might have been ID 4891 and is now ID 72145. The model still looks up row 4891 for that text, but row 4891 now belongs to some other token. So every lookup pulls the wrong learned vector, and the forward pass produces nonsense.
There is a second component that breaks the same way. The output head, `lm_head`, has shape `[d_model, vocab_size]`. Going from 50k to 100k tokens means 50,000 new output rows with no trained weights, so the model cannot produce those new tokens sensibly.
The fixes range from cheap to expensive: keep the old IDs and only append new ones, or initialize the new rows by averaging their subword pieces and continue training, or retrain the whole model from scratch. The reason this is a hard question is that tokenizer choice is effectively locked once the model is pretrained.
Two model components carry `vocab_size` as a dimension, and both break when the vocabulary changes.
The input embedding table has shape `[vocab_size, d_model]`, one learned row per token ID. The output projection `lm_head` has the mirror shape `[d_model, vocab_size]`. Growing from 50k to 100k tokens reshapes both.
The first failure is **ID remapping**. "hello" was ID 4891, now it is 72145, but the model still fetches row 4891 for that text. Every embedding lookup returns the wrong learned vector, so the forward pass is garbage. The second failure is on the output side: the 50,000 new rows of the head have never been trained.
The remediation ladder has three rungs. Vocabulary expansion keeps existing IDs intact, appends the new ones, and trains only the new rows; this works when you extend rather than replace. Vocabulary adaptation initializes each new token's embedding from the average of its old subword pieces, then runs continued pretraining. Full retraining is correct but pays the entire pretraining bill.
The senior point is that tokenizer choice is sticky: it is fixed at pretraining time, so a multilingual upgrade is a training project, not a config change.
Quick reference
TL;DR
Swapping vocabularies misaligns the embedding table and lm_head, both indexed by token ID, so every lookup and logit points at the wrong token; fixes range from expansion to adaptation to full retrain.
Imagine you wrote a phone book where each name sits at a numbered line, and you memorized which name lives on which line. Now a friend hands you a new phone book with twice the names, in a totally different order. Your memorized line numbers all point to the wrong people now. The model has the same kind of memory: it learned what lives at each numbered line, the tokenizer is what assigns those line numbers, and a new tokenizer reshuffles the lines. You either rewrite the whole memory from scratch, or carefully copy what you can and learn the new entries, but you cannot just trust the old line numbers anymore.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Choosing a tokenizer swap as the fix for poor multilingual coverage on a model already in production The embedding and head weights are indexed by the original IDs, so a swap misaligns every learned row and the only correct routes are adaptation or a full retrain.
Adapting a model when the target requires entirely new linguistic knowledge, not just new tokens A few thousand new embedding rows cannot carry deep knowledge of an unseen language, so vocabulary work alone is insufficient and you need domain pretraining.
Replacing the full vocabulary to chase a lower fertility number on a small budget Replacement invalidates every existing row simultaneously, so the continued-training cost approaches a fresh pretraining run and the savings vanish.
Expanding the vocabulary by inserting new IDs in the middle of the existing range Inserting shifts every later ID, misaligning the trained rows; only appending at the end keeps the old mapping and weights valid.
Quick scan · 2–4 min
Everything important, quickly.
4 min: two tensors carry vocab_size + remapping misaligns old rows + expansion adds untrained rows + expand vs adapt vs retrain + subword-average init + why the choice is sticky.
Real-world examples
Real products, models, and research that use this idea.
1When OpenAI introduced o200k_base for GPT-5.5, it came with new model training rather than a tokenizer swap into prior checkpoints, because the embedding and head weights are bound to the original IDs.
2Open-source teams localizing Llama 4 to Indic or Arabic scripts use embedding initialization by averaging plus continued pretraining, not a tokenizer replacement, to add coverage.
3Hugging Face's resize_token_embeddings API supports vocabulary expansion by appending rows, which is why adding special tokens works but wholesale vocabulary replacement does not.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you must keep the same model and only add 5,000 domain tokens, what exact steps preserve existing behavior?
Sign in or go Pro to see the approach for this follow-up.
QHow would you measure whether vocabulary adaptation succeeded before committing to a full deployment?
Sign in or go Pro to see the approach for this follow-up.
QWhat goes wrong if you initialize the new embedding rows randomly instead of from subword averages?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Listing only the embedding table while forgetting that the lm_head shares the vocab_size dimension and breaks identically on the output side.