The right answer is **B: reuse the base tokenizer when fine-tuning an instruction-tuned model; train a fresh BPE only when pretraining from scratch**.
The model's tokenizer is welded to its embedding matrix and lm_head. Each token id corresponds to a specific row in those tensors that was trained over trillions of pretraining tokens. If you swap the tokenizer, the new ids point to embedding rows that have no learned meaning. You would have to retrain the embeddings from scratch, which is what pretraining does. For a fine-tune (typically 100M to a few B tokens, much less than pretraining), there is no chance of recovering the lost capability.
So when you are fine-tuning Llama 3, GPT-5, or any other pretrained model: keep the original tokenizer. Yes, code fertility is somewhat higher than it could be (Python whitespace and operators are not optimally compressed), but the trade is worth it because every other capability the model has remains intact.
When you are pretraining a brand-new code model from scratch: train a fresh BPE on a large code corpus. Dedicated tokens for indentation, common identifiers (`def`, `return`, `range`), and operators drop fertility materially. Codestral, Qwen Coder 2.5, and GPT-5-code take this approach for the parts that are pretrained from scratch, with careful pre-tokenization that breaks code into sensible chunks before BPE.
Vocab extension (C) is the middle option and is only worth it when you have a clear gap and enough fine-tune data to train the new embeddings without breaking the rest.
**B: reuse the base tokenizer when fine-tuning, fresh BPE only when pretraining.**
The key insight: tokenizer and pretrained weights are one trained artifact. The embedding matrix has 128K to 256K rows, each carefully positioned in semantic space by trillions of pretraining tokens. The lm_head mirrors this. Swap the tokenizer and the new ids index into rows with no learned meaning, effectively wiping every word the model knows.
**When fine-tuning an instruction-tuned model:** reuse the base tokenizer. Yes, code fertility on Python is suboptimal (Llama 3's BPE has decent code coverage but not great whitespace tokens). The trade is worth it: you keep the pretrained embeddings, the instruction tuning, and the entire latent capability. Pre-tokenization tricks at training time can still help the model handle code well without changing the tokenizer.
**When pretraining from scratch:** train a fresh code-aware BPE. Dedicated tokens for indentation levels, common Python keywords (`def`, `return`, `import`, `self`, `range`), operators, and string formatting drop fertility 20 to 40 percent versus a general-purpose tokenizer. Codestral, Qwen Coder 2.5, DeepSeek-Coder-V2, and GPT-5-code take this approach for their code-pretrained components, often with careful pre-tokenization regex that respects whitespace and operator boundaries.
**Extension (option C)** is the right move only when (1) you have a clear vocabulary gap in the base tokenizer for your domain, (2) you have enough fine-tune data to train the new embedding rows (rule of thumb: 100+ occurrences per new token), and (3) you accept some degradation of the base instruction-tuning. For pure Python fine-tunes on top of an already code-competent base, extension rarely earns its cost.
**Option D** (skip the question) is wrong because the model cannot internally adapt embeddings the tokenizer never produces; the tokenizer determines what the embedding layer can possibly see.
Quick reference
TL;DR
Fine-tuning an instruction-tuned model: reuse the base tokenizer. Pretraining from scratch: train a fresh code-aware BPE. Extension is a middle ground only worth it with a clear vocab gap and enough fine-tune data.
Imagine you have a friend who already speaks English and writes neat handwriting. You want to teach them to be better at writing Python. Option one is to keep the alphabet they already know and just have them practice Python writing more. Option two is to invent a new alphabet just for Python. Option three is to add new letters to their existing alphabet for special Python words. If you switch alphabets entirely, your friend forgets how to write English the next day. If you only add a few new letters and have them practice a lot, they keep their English and pick up Python tricks. If you are starting from a baby who has no alphabet yet, inventing a Python-only one from the start makes sense. The right answer depends on whether your friend already knows a lot or is starting fresh.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Fine-tuning a base model (not instruction-tuned) on a very large new-domain corpus where embedding retraining is feasible. At pretraining-scale data on a base model, the constraints look more like fresh-pretraining and tokenizer replacement becomes viable; the decision tree above assumes typical fine-tune data volumes.
Adapting an already code-pretrained model like Codestral to a sub-domain of code (Python-only from a polyglot base). The base tokenizer is already strong on code; fertility wins from further specialization are small and rarely justify the engineering cost.
Building a research model where reproducibility and clean tokenizer choices matter more than production efficiency. Research contexts may prioritize a clean controlled experiment over the practical reuse versus replace tradeoff that production cares about.
Quick scan · 2–4 min
Everything important, quickly.
Open with the framing that tokenizer and model embeddings are one trained artifact. Branch the decision: fine-tuning an instruction-tuned model means reuse the base tokenizer; pretraining from scratch means train a fresh code-aware BPE. Cover the extension middle option with its preconditions (clear vocab gap, ~100 occurrences per new token, tolerance for some instruction-tuning degradation). Cite Codestral, Qwen Coder 2.5, and GPT-5-code as the 2026 production examples.
Real-world examples
Real products, models, and research that use this idea.
1Codestral, Qwen Coder 2.5, and DeepSeek-Coder-V2 train their code-pretrained components with fresh code-aware BPE that has dedicated tokens for indentation and common identifiers.
2Code Llama and Code Llama Python kept the base Llama 2 tokenizer and added a small set of code-specific tokens via vocab extension, demonstrating the middle path on a constrained data budget.
3HuggingFace's `tokenizers` library exposes a pre_tokenizer pipeline that lets you customize whitespace and operator handling without retraining the BPE, useful for code fine-tunes on a base model.
4tiktoken's cl100k_base and o200k_base both include code-aware merges that handle common Python and JavaScript reasonably, even though they were trained on a broader corpus.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow much fine-tune data does it take to make vocab extension actually pay off?
Sign in or go Pro to see the approach for this follow-up.
QIf reuse hurts code fertility on Llama 3, what non-tokenizer fixes can recover some of the cost?
Sign in or go Pro to see the approach for this follow-up.
QWhy do Codestral and DeepSeek-Coder train a fresh code BPE rather than extend a base model's vocab?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Replacing the tokenizer when fine-tuning an instruction-tuned model, then being surprised the model lost all its pretrained capability.