You are fine-tuning on Python code only. Should you reuse the base tokenizer, train a new one, or extend the vocab?

MCQ·Medium·4.0 · 0·~1 min·Asked atLtimindtree OpenAI Tencent·Relevant atMeta

Attempt it

The right answer is **B: reuse the base tokenizer when fine-tuning an instruction-tuned model; train a fresh BPE only when pretraining from scratch**. The model's tokenizer is welded to its embedding matrix and lm_head. Each token id corresponds to a specific row in those tensors that was trained over trillions of pretraining tokens. If you swap the tokenizer, the new ids point to embedding rows that have no learned meaning. You would have to retrain the embeddings from scratch, which is what pretraining does. For a fine-tune (typically 100M to a few B tokens, much less than pretraining), there is no chance of recovering the lost capability. So when you are fine-tuning Llama 3, GPT-5, or any other pretrained model: keep the original tokenizer. Yes, code fertility is somewhat higher than it could be (Python whitespace and operators are not optimally compressed), but the trade is worth it because every other capability the model has remains intact. When you are pretraining a brand-new code model from scratch: train a fresh BPE on a large code corpus. Dedicated tokens for indentation, common identifiers (`def`, `return`, `range`), and operators drop fertility materially. Codestral, Qwen Coder 2.5, and GPT-5-code take this approach for the parts that are pretrained from scratch, with careful pre-tokenization that breaks code into sensible chunks before BPE. Vocab extension (C) is the middle option and is only worth it when you have a clear gap and enough fine-tune data to train the new embeddings without breaking the rest.

**B: reuse the base tokenizer when fine-tuning, fresh BPE only when pretraining.** The key insight: tokenizer and pretrained weights are one trained artifact. The embedding matrix has 128K to 256K rows, each carefully positioned in semantic space by trillions of pretraining tokens. The lm_head mirrors this. Swap the tokenizer and the new ids index into rows with no learned meaning, effectively wiping every word the model knows. **When fine-tuning an instruction-tuned model:** reuse the base tokenizer. Yes, code fertility on Python is suboptimal (Llama 3's BPE has decent code coverage but not great whitespace tokens). The trade is worth it: you keep the pretrained embeddings, the instruction tuning, and the entire latent capability. Pre-tokenization tricks at training time can still help the model handle code well without changing the tokenizer. **When pretraining from scratch:** train a fresh code-aware BPE. Dedicated tokens for indentation levels, common Python keywords (`def`, `return`, `import`, `self`, `range`), operators, and string formatting drop fertility 20 to 40 percent versus a general-purpose tokenizer. Codestral, Qwen Coder 2.5, DeepSeek-Coder-V2, and GPT-5-code take this approach for their code-pretrained components, often with careful pre-tokenization regex that respects whitespace and operator boundaries. **Extension (option C)** is the right move only when (1) you have a clear vocabulary gap in the base tokenizer for your domain, (2) you have enough fine-tune data to train the new embedding rows (rule of thumb: 100+ occurrences per new token), and (3) you accept some degradation of the base instruction-tuning. For pure Python fine-tunes on top of an already code-competent base, extension rarely earns its cost. **Option D** (skip the question) is wrong because the model cannot internally adapt embeddings the tokenizer never produces; the tokenizer determines what the embedding layer can possibly see.

TL;DR

Fine-tuning an instruction-tuned model: reuse the base tokenizer. Pretraining from scratch: train a fresh code-aware BPE. Extension is a middle ground only worth it with a clear vocab gap and enough fine-tune data.

Memory aid

Easy to grasp

Imagine you have a friend who already speaks English and writes neat handwriting. You want to teach them to be better at writing Python. Option one is to keep the alphabet they already know and just have them practice Python writing more. Option two is to invent a new alphabet just for Python. Option three is to add new letters to their existing alphabet for special Python words. If you switch alphabets entirely, your friend forgets how to write English the next day. If you only add a few new letters and have them practice a lot, they keep their English and pick up Python tricks. If you are starting from a baby who has no alphabet yet, inventing a Python-only one from the start makes sense. The right answer depends on whether your friend already knows a lot or is starting fresh.

Key concepts

Code tokenizer Fine-tuning Vocab extension

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The question 'should I train a code-specific tokenizer?' is one of the most common decision points in domain fine-tuning, and the answer is conditional in ways many engineers miss. The right framing is that the tokenizer is part of the model: the embedding matrix and lm_head are indexed by token ids, and those rows were positioned in semantic space by trillions of pretraining tokens. Swap the tokenizer and you swap which rows the model reads, which is equivalent to wiping the model's vocabulary knowledge.

The decision branches on what you are doing. Pretraining a new model from scratch frees you to train a fresh code-aware BPE that drops fertility materially. Fine-tuning an instruction-tuned model constrains you to reuse the base tokenizer because there is no path to retraining embeddings at fine-tune scale. Extension sits between the two as a middle option that is worth it only under specific preconditions.

The rest of this explanation walks each branch in detail, names what 2026 production code models actually do, and explains why option D (skip the question) is wrong because the tokenizer determines what the embeddings can possibly see.

Why the tokenizer is part of the model

Modern transformer models have three pieces tightly coupled to their tokenizer: the input embedding matrix (one row per token id, vocab_size by d_model), the lm_head (often tied to the input embedding), and the model's learned attention patterns over specific token sequences. All three were shaped by pretraining on whatever the tokenizer produced.

The embedding row for token id 42 is not a generic 'word 42 vector'. It is a specific position in semantic space that the model learned over many billions of training updates. The same id under a different tokenizer would point to a different byte sequence, and the model would have no learned representation for it. Loading a Llama 3 model and feeding it ids from a different tokenizer is functionally equivalent to randomizing the embedding lookup, which is what 'wiping the model's vocabulary' means.

This coupling is asymmetric. You can change the model's weights during fine-tuning, because gradient descent updates them based on a loss signal. You cannot change the tokenizer the same way; the tokenizer is a discrete mapping from bytes to ids with no gradient to apply. So tokenizer-level changes have to be either (1) replacement plus full re-pretraining, (2) extension plus targeted embedding training, or (3) reuse with no change.

Fine-tuning: why reuse is the only safe path

Pretraining from scratch: when fresh BPE wins

Extension as the middle option and its preconditions

Why option D is wrong: tokenizer determines what embeddings can see

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Codestral, Qwen Coder 2.5, and DeepSeek-Coder-V2 train their code-pretrained components with fresh code-aware BPE that has dedicated tokens for indentation and common identifiers.
Code Llama and Code Llama Python kept the base Llama 2 tokenizer and added a small set of code-specific tokens via vocab extension, demonstrating the middle path on a constrained data budget.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow much fine-tune data does it take to make vocab extension actually pay off?

Heuristic: 100+ occurrences of each new token in the fine-tune corpus, ideally several hundred. For 1,000 new Python tokens at 100 occurrences each, you need ~100K well-distributed instances of those tokens in your fine-tune data, typically a few GB of code. Below that, the new embeddings under-train and the model effectively falls back to bytes through them anyway.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Replacing the tokenizer when fine-tuning an instruction-tuned model, then being surprised the model lost all its pretrained capability.

60 second bullets to scan on the way to the call.

Why the tokenizer is part of the model and not freely swappable.
When fresh BPE is appropriate (pretraining only).

Primary sources. Browse if you want the original framing.