A team has a pretrained LLM checkpoint trained with a 50,000 token vocabulary. They want to swap to a new tokenizer with a 100,000 token vocabulary that has better multilingual coverage. Explain precisely what breaks at the model architecture and weight level, and describe what remediation options exist with their costs.
Swapping vocabularies misaligns the embedding table and lm_head, both indexed by token ID, so every lookup and logit points at the wrong token; fixes range from expansion to adaptation to full retrain.
Imagine you wrote a phone book where each name sits at a numbered line, and you memorized which name lives on which line. Now a friend hands you a new phone book with twice the names, in a totally different order. Your memorized line numbers all point to the wrong people now. The model has the same kind of memory: it learned what lives at each numbered line, the tokenizer is what assigns those line numbers, and a new tokenizer reshuffles the lines. You either rewrite the whole memory from scratch, or carefully copy what you can and learn the new entries, but you cannot just trust the old line numbers anymore.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This is a senior-level question because the naive instinct, that a tokenizer is a preprocessing detail you can swap freely, is exactly wrong, and the interviewer wants to see whether you know why. A model does not consume text. It consumes integer token IDs, and every learned parameter assumes a fixed mapping between string pieces and those integers.
The answer has two halves that candidates often blur together. There is the misalignment problem, where existing token IDs point at the wrong learned rows after a remap, and there is the expansion problem, where a bigger vocabulary adds rows that were never trained. They have different mechanisms and different fixes. We will separate them, attach the affected tensors, walk the remediation ladder with its costs, and close on why the whole thing is effectively irreversible.
The two tensors that depend on vocabulary size
Only two tensors in a transformer carry vocab_size as a dimension. The input embedding table is [vocab_size, d_model]: it maps each token ID to a learned d_model-dimensional vector. The output projection, the language-model head, is [d_model, vocab_size]: it maps the final hidden state to one logit per possible next token.
Everything in between, the attention layers and feed-forward blocks, operates purely on d_model vectors and is blind to the vocabulary. So a vocabulary change cannot corrupt attention weights; it only touches the entry and exit of the model.
That is the precise frame for the question. The tokenizer defines the index into these two tensors, and the tensors were trained against one specific index. Change the index and you have invalidated the trained contents of both, on the input side and the output side at once.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- When OpenAI introduced o200k_base for GPT-5.5, it came with new model training rather than a tokenizer swap into prior checkpoints, because the embedding and head weights are bound to the original IDs.
- Open-source teams localizing Llama 4 to Indic or Arabic scripts use embedding initialization by averaging plus continued pretraining, not a tokenizer replacement, to add coverage.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you must keep the same model and only add 5,000 domain tokens, what exact steps preserve existing behavior?
Expansion: append the 5,000 IDs at the end of embedding and lm_head, initialize from subword averages, freeze old rows, and train only the new rows on domain text.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Listing only the embedding table while forgetting that the lm_head shares the vocab_size dimension and breaks identically on the output side.
60 second bullets to scan on the way to the call.
The two tensors whose shape includes vocab_size
ID remapping versus row expansion as separate failures
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.