You're extending a Llama-3 tokenizer with 500 new domain-specific tokens (pharmaceutical compound names, ICD-10 codes, etc.) before fine-tuning. How do you initialise the embeddings for the new tokens, and why is random initialisation a bad idea?
Free · 2 AI evals / day
When you add new tokens, you grow the embedding table by 500 rows. The question is what numbers to put in those fresh rows before training.
The existing rows are not random. After pretraining, every token's vector sits in a specific region, and the rest of the model is tuned to read vectors from that region. If you fill the new rows with random Gaussian noise, they land in the wrong region entirely.
At the first training step, any prompt with a new token then produces strange internal activations. The gradients become huge and jumpy, and the run wastes its early steps just cleaning up the mess you injected.
The fix is cheap. Each new token would have been split into smaller subword pieces by the old tokenizer. Those pieces already have good vectors, so average them and use that mean as the starting point. The new token now lands in a sensible neighbourhood.
Do the same for the output side of the model, then fine-tune so the model truly learns the new tokens rather than just tolerating them.
Never initialise new token rows from random noise. The pretrained embedding table occupies a calibrated region, and the downstream layers expect inputs from that region. Random vectors land off this manifold.
The consequence shows up at step 0. Prompts containing a new token produce off-distribution activations. Gradients spike, loss is unstable, and the optimiser burns its early budget reabsorbing the noise rather than learning.
The standard recipe is the mean of subwords init. Resize the model so the table grows by 500 rows. For each new token, find the subword pieces it would have decomposed into under the original tokenizer, then set its embedding to the mean of those pieces. The new token starts near its semantic parts.
Treat both matrices. The input embedding gets this init, and so does the output unembedding matrix, whether tied or separate. The output rows need a meaningful logit at step 0 too.
A stronger variant uses semantic anchors. If you have a list of related in-domain tokens, average those instead for a closer start.
Initialisation is only a head start. You still must train the new rows. With PEFT, set modules_to_save to embed_tokens and lm_head, because LoRA adapters alone never touch the embedding table.
Quick reference
TL;DR
Resize the embedding matrix, then initialise each new token as the mean of the subword pieces it used to split into. Treat input and output rows. Then train.
Picture a huge map where every word the model knows sits at a meaningful spot, and nearby spots mean similar things. When you invent a brand-new word, you need to drop it somewhere on this map. Dropping it at a random spot is like teleporting a stranger into the middle of the ocean: the model has no idea what it means, and its first guesses are wild. A smarter move is to look at the smaller word-pieces the new word used to be chopped into, find where each piece already sits, and place the new word at the average of those spots. Now it lands in a sensible neighbourhood from the very first step, and training only has to nudge it, not rescue it.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
The new strings are already covered well by existing subword tokens with no efficiency gain Adding vocabulary buys nothing here and forces embedding training. Keep the original tokenizer instead.
You only need a handful of rare phrases that appear a few times total Those tokens never get enough gradient signal to train, so the new rows stay essentially at their init.
The base model ties input and output embeddings and your code overwrites only one A partial overwrite leaves the tied matrix inconsistent, so verify the tie before applying the mean init.
You are doing pure LoRA with frozen embeddings for a quick behaviour tweak If you cannot afford to train embed_tokens, do not extend the vocabulary at all in this run.
Quick scan · 2–4 min
Everything important, quickly.
5 min: why random init is off-distribution + step-0 instability + mean of subwords recipe + treating the output head + semantic-neighbour variant + training the rows with modules_to_save.
Code example
python
import torch
model.resize_token_embeddings(len(tokenizer))
emb = model.get_input_embeddings().weight.data
for new_id, surface in new_tokens.items():
pieces = base_tokenizer(surface, add_special_tokens=False)['input_ids']
emb[new_id] = emb[pieces].mean(dim=0)
# mirror onto the output head if untied
out = model.get_output_embeddings().weight.data
for new_id, surface in new_tokens.items():
pieces = base_tokenizer(surface, add_special_tokens=False)['input_ids']
out[new_id] = out[pieces].mean(dim=0)
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face Transformers exposes resize_token_embeddings, which grows both embed_tokens and lm_head before you overwrite the new rows with a mean init.
2PEFT's modules_to_save=['embed_tokens', 'lm_head'] is the documented way to train new vocabulary rows alongside LoRA adapters when adapting Llama 4 or Qwen3.
3Domain models like BioMedLM and Code Llama extended tokenizers for biomedical and code vocabularies, where mean of subwords style init stabilised early training.
4Teams adapting GPT-OSS or DeepSeek V4 for ICD-10 and pharma corpora seed new compound-name tokens from averaged related domain tokens rather than random vectors.
5Mistral and Gemma fine-tuning recipes on Together.ai and Modal Labs apply mean-init to added tokens before LoRA to avoid loss spikes at step 0.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does an off-distribution output (unembedding) row hurt training even more sharply than a bad input row?
Sign in or go Pro to see the approach for this follow-up.
QWhen would semantic-neighbour init clearly beat mean of subwords, and what does it cost you?
Sign in or go Pro to see the approach for this follow-up.
QHow would you confirm the new tokens actually trained rather than staying near their init?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Initialising new rows from random Gaussian noise, then wondering why loss spikes. The rows sit off the embedding manifold, so step-0 activations and gradients are unstable.