Adding 500 domain tokens to the tokenizer: how do you initialise their embeddings?
You're extending a Llama-3 tokenizer with 500 new domain-specific tokens (pharmaceutical compound names, ICD-10 codes, etc.) before fine-tuning. How do you initialise the embeddings for the new tokens, and why is random initialisation a bad idea?
Resize the embedding matrix, then initialise each new token as the mean of the subword pieces it used to split into. Treat input and output rows. Then train.
Picture a huge map where every word the model knows sits at a meaningful spot, and nearby spots mean similar things. When you invent a brand-new word, you need to drop it somewhere on this map. Dropping it at a random spot is like teleporting a stranger into the middle of the ocean: the model has no idea what it means, and its first guesses are wild. A smarter move is to look at the smaller word-pieces the new word used to be chopped into, find where each piece already sits, and place the new word at the average of those spots. Now it lands in a sensible neighbourhood from the very first step, and training only has to nudge it, not rescue it.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: why random init is off-distribution + step-0 instability + mean of subwords recipe + treating the output head + semantic-neighbour variant + training the rows with modules_to_save.
import torch
model.resize_token_embeddings(len(tokenizer))
emb = model.get_input_embeddings().weight.data
for new_id, surface in new_tokens.items():
pieces = base_tokenizer(surface, add_special_tokens=False)['input_ids']
emb[new_id] = emb[pieces].mean(dim=0)
# mirror onto the output head if untied
out = model.get_output_embeddings().weight.data
for new_id, surface in new_tokens.items():
pieces = base_tokenizer(surface, add_special_tokens=False)['input_ids']
out[new_id] = out[pieces].mean(dim=0)Real products, models, and research that use this idea.
- Hugging Face Transformers exposes resize_token_embeddings, which grows both embed_tokens and lm_head before you overwrite the new rows with a mean init.
- PEFT's modules_to_save=['embed_tokens', 'lm_head'] is the documented way to train new vocabulary rows alongside LoRA adapters when adapting Llama 4 or Qwen3.
- Domain models like BioMedLM and Code Llama extended tokenizers for biomedical and code vocabularies, where mean of subwords style init stabilised early training.
- Teams adapting GPT-OSS or DeepSeek V4 for ICD-10 and pharma corpora seed new compound-name tokens from averaged related domain tokens rather than random vectors.
- Mistral and Gemma fine-tuning recipes on Together.ai and Modal Labs apply mean-init to added tokens before LoRA to avoid loss spikes at step 0.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does an off-distribution output (unembedding) row hurt training even more sharply than a bad input row?
QWhen would semantic-neighbour init clearly beat mean of subwords, and what does it cost you?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Initialising new rows from random Gaussian noise, then wondering why loss spikes. The rows sit off the embedding manifold, so step-0 activations and gradients are unstable.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.