**Option B is correct.** A tokenizer is a deterministic algorithm that splits raw text into subword pieces and assigns each piece a fixed integer ID. The IDs come from a vocabulary file built when the base model was pretrained. Common tokenizer algorithms include BPE, WordPiece, and SentencePiece Unigram.
Those integer IDs are not random labels; they are row indices into the model's embedding matrix. If the tokenizer says the word 'cat' becomes ID 12345, the model looks up row 12345 of the embedding matrix to get the vector that represents 'cat'.
The constraint that follows: the tokenizer used at fine-tuning and at inference must be exactly the same one the base model was pretrained with. If you swap it, the same text encodes to different IDs, those IDs index into the same embedding matrix, and the model retrieves the wrong vectors for every token. Outputs become garbled almost immediately.
The other three options describe things that are not true. The tokenizer is not a neural network, it is not an optional runtime step, and it does not produce embeddings directly.
**Answer B.** A tokenizer is a fixed, pre-trained, non-neural algorithm that maps text to integer IDs. The IDs are row indices into the model's embedding matrix, so the integer 12,345 means 'fetch row 12,345 of the embedding table'.
The tokenizer is shipped with the base model and is bound to it. The vocabulary file lists every subword token and its assigned ID; the algorithm (BPE, WordPiece, SentencePiece Unigram) is what produced those IDs from the original pretraining corpus. You can read it as a frozen lookup table.
The consequence for fine-tuning is strict: use the same tokenizer end to end. The base model's embedding matrix has 32,000 (or 128,000, or whatever vocabulary size) specific rows arranged in a specific order. A different tokenizer would produce different IDs for the same text, and the embedding lookup would pull rows that mean something entirely unrelated. Quality collapses immediately.
The distractors capture common misconceptions. Option A treats the tokenizer as trainable, which it is not, gradients do not flow into the tokenizer. Option C calls it optional, which it never is. Option D conflates the tokenizer with the embedding layer; the tokenizer produces integers, the embedding layer turns those integers into vectors.
Quick reference
TL;DR
A tokenizer is a fixed text to integer mapping whose IDs index the embedding matrix. The same tokenizer used at pretraining must be reused at fine-tuning and inference, or the embedding lookup pulls the wrong rows.
Picture a library where every book has a shelf number. The tokenizer is the librarian who tells you which shelf a request goes to. The shelves themselves are the model's giant lookup table of word meanings. If a new librarian shows up using a different numbering system but the shelves stay the same, asking for the cookbook now retrieves a math textbook. The shelves did not move; the numbers point somewhere else. That is what happens when you swap the tokenizer between training and serving. The text looks the same, but the integer IDs change, the lookup retrieves the wrong rows of meanings, and nothing the model says afterward makes sense.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You genuinely need a different tokenizer for a different language or domain, like a Hebrew-specific tokenizer for a Hebrew fine-tune Swapping tokenizers requires retraining the embedding matrix from scratch (or extensive continued pretraining), which is a much larger project than a typical fine-tune.
You want to add a small set of domain-specific control tokens without disturbing the base vocabulary Use add_tokens plus resize_token_embeddings and train the new rows, do not swap the whole tokenizer; the rest of the vocabulary keeps its meaning.
You are debugging a fine-tune that produces garbled output starting from step zero The first thing to check is whether the tokenizer matches the base model, because mismatched tokenizers are the single most common cause of this exact symptom.
Quick scan · 2–4 min
Everything important, quickly.
3 min: state the tokenizer as a fixed text to integer mapping, explain that IDs index the embedding matrix, name the strict end to end matching constraint, and end with the extend vs replace distinction.
Real-world examples
Real products, models, and research that use this idea.
1Llama 3.1 ships with a tiktoken-based BPE tokenizer of 128,256 tokens; fine-tunes that swap it for a different vocabulary produce immediately broken outputs.
2DeepSeek V4 uses a custom BPE tokenizer with byte-fallback for unseen characters; mixing it with a Mistral tokenizer at serving time corrupts every Cyrillic and CJK input.
3Hugging Face's AutoTokenizer.from_pretrained pulls the exact tokenizer files (vocab.json, merges.txt, tokenizer.json) bound to a model checkpoint to avoid this whole class of bug.
4Claude Opus 4.7 and Gemini 3.1 Pro both publish tokenizer details so wrappers can compute token counts accurately for billing and context length.
5Adding custom domain tokens for medical fine-tunes typically uses tokenizer.add_tokens followed by model.resize_token_embeddings, with the new embedding rows placed in modules_to_save for the LoRA run.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through what happens when you call tokenizer.add_tokens followed by model.resize_token_embeddings, and why both calls are required.
Sign in or go Pro to see the approach for this follow-up.
QWhat is byte-fallback in a BPE or SentencePiece tokenizer, and why does it matter for fine-tuning on non-English text?
Sign in or go Pro to see the approach for this follow-up.
QHow would you detect a silent tokenizer mismatch in production, and what is the cheapest sanity check at deployment time?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating the tokenizer as a trainable component or as an optional preprocessing detail. It is a fixed mapping that has to match end to end, and forgetting that is one of the most common causes of broken fine-tunes.