You want to add 5,000 Hindi medical terms to Llama 3 8B. Walk through the procedure and the risks.

Short answer·Medium·4.0 · 0·~3 min·Relevant atMeta Sarvam

Attempt it

Free · 2 AI evals / day

Adding tokens to a fine-tuned model has five steps and most teams skip at least one of them. **Step 1: choose the new tokens.** Train a small BPE on your Hindi medical corpus (5 to 10 GB), then take the top merges that are not already in Llama 3's 128K vocabulary. Cap the additions at your target (5,000). This is data-driven; hand-picking tokens is a worse choice because BPE is what the rest of the model expects. **Step 2: extend the tokenizer.** `tokenizer.add_tokens(new_tokens)`. This bumps the vocab size and assigns ids 128,256 through 133,255 to the new tokens. **Step 3: resize the model.** `model.resize_token_embeddings(len(tokenizer))`. This grows both the input embedding matrix and the lm_head to the new vocab size. HuggingFace resizes both together when they are tied. The new rows start as random. **Step 4: better initialization.** Random embeddings are gibberish until fine-tuning trains them. Better: initialize each new token's row as the mean of the embeddings of the subwords the token would have decomposed to under the old tokenizer. This gives the model a sensible starting point. **Step 5: fine-tune and save.** Fine-tune on enough Hindi medical data that the new embeddings learn meaningful representations. Save the tokenizer alongside the model with `tokenizer.save_pretrained` so inference loads the matching vocab. **Failure modes.** Catastrophic forgetting of English if you fine-tune Hindi-only; mix in 10 to 30 percent English. Under-trained additions produce gibberish at inference. Forgetting to save the tokenizer means inference loads the base 128K vocab and your new embedding rows are unreachable.

**Five-step procedure.** 1. **Select tokens via domain BPE.** Train a small BPE on 5 to 10 GB of Hindi medical text. Take the top merges not already in Llama 3's 128K vocabulary. Cap at 5,000. Data-driven selection beats hand-picking. 2. **Extend the tokenizer.** `tokenizer.add_tokens(new_tokens)`. New ids start at 128,256. 3. **Resize the model.** `model.resize_token_embeddings(len(tokenizer))` grows the input embedding matrix and the lm_head. HuggingFace handles both together when weights are tied; verify on your model class. 4. **Embedding-mean init.** Random init means the new tokens produce gibberish until trained. Better: for each new token, decompose it under the old tokenizer, take the embeddings of those subwords, average them. Same logic for lm_head rows. 5. **Fine-tune and save.** Under-trained additions still produce gibberish. After training, `tokenizer.save_pretrained(path)` alongside `model.save_pretrained(path)`. **Risks.** Catastrophic forgetting of English if Hindi-only; mix 10 to 30 percent English. Skipping the lm_head resize leaves the model unable to predict new ids. Mean-init fails for tokens with no clean subword decomposition; fall back to random for those. Forgetting the tokenizer save means inference uses the base 128K vocab and fine-tuned rows are unreachable. Vocab extension breaks some base instruction-tuned behavior the fine-tune has to compensate for.

Quick reference

TL;DR

Train BPE on Hindi, add_tokens, resize_token_embeddings (input AND lm_head), mean-init new rows, fine-tune with English mixed in, save the tokenizer. Watch forgetting and under-trained tokens.

Memory aid

Easy to grasp

Imagine teaching a friend who reads English to read Hindi medical reports. You decide to give them index cards for 5,000 common Hindi medical words so they do not have to spell each one letter by letter. The procedure has five parts. Pick the right 5,000 words by looking at lots of real Hindi medical text, not by guessing. Add the index cards to their card box and make sure their brain has space for the new words on both the 'reading' side and the 'speaking' side. Write a sensible first guess on each new card based on the spelling parts your friend already knows. Have them practice with enough Hindi medical text to actually learn the cards. Finally, give them the updated card box every time they go to read, not the old one. Skip any step and you get a friend who has forgotten some English, makes up nonsense for the new words, or cannot find the new cards at all.

Key concepts

Vocabulary Extension Fine-tuning Embedding Initialization

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Walk the five steps in order: domain-BPE selection, add_tokens, resize_token_embeddings (input plus lm_head), embedding-mean init, then fine-tune and save. Name catastrophic forgetting and under-trained embeddings as the two main failure modes. Cover the parameter cost (about 80M params for 5K tokens on 8B model) and the tokenizer.save_pretrained discipline. Close with the data-volume heuristic: at least 100 occurrences per new token for it to learn well.

Code example

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tok = AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-8B')
model = AutoModelForCausalLM.from_pretrained('meta-llama/Meta-Llama-3-8B')

# 1. Select via domain BPE (sketch)
new_tokens = train_hindi_medical_bpe(corpus_path)[:5000]
new_tokens = [t for t in new_tokens if t not in tok.get_vocab()]

# 2. Add to tokenizer
num_added = tok.add_tokens(new_tokens)

# 3. Resize model (handles both input embedding and lm_head)
model.resize_token_embeddings(len(tok), pad_to_multiple_of=8)

# 4. Embedding-mean init for new rows
emb = model.get_input_embeddings().weight
lm_head = model.get_output_embeddings().weight
for i, tok_str in enumerate(new_tokens):
    sub_ids = tok.encode(tok_str, add_special_tokens=False)
    # Use the OLD subword decomposition for init
    new_id = len(tok) - num_added + i
    with torch.no_grad():
        emb[new_id] = emb[sub_ids].mean(dim=0)
        lm_head[new_id] = lm_head[sub_ids].mean(dim=0)

# 5. Fine-tune, then save tokenizer alongside model
model.save_pretrained(out_path)
tok.save_pretrained(out_path)  # critical

Real-world examples

Real products, models, and research that use this idea.

OpenHathi (Sarvam AI) and Project Indus extended Llama 2 with Hindi tokens before fine-tuning on Hindi corpora; the published procedure uses BPE-trained additions plus mean init and explicitly warns about catastrophic forgetting.
HuggingFace transformers ships resize_token_embeddings with a pad_to_multiple_of parameter to keep new embedding matrices tensor-core aligned.
Meta's Code Llama and Code Llama Python were trained from Llama 2 with extended code-aware tokenization, using the same pattern of BPE-trained additions on top of the base vocab.
Mistral and Qwen 3.5 multilingual variants ship with extended vocabularies for Devanagari, CJK, and Arabic, reducing fertility on those scripts compared to the base.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow many fine-tune tokens do you need before the new embedding rows are competitive with byte-fallback through the old tokenizer?

QShould you use LoRA or full fine-tune when extending vocab?

QWhat changes if you want to add tokens to an instruction-tuned model rather than a base model?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Calling add_tokens without resize_token_embeddings, or resizing only the input embedding while leaving the lm_head unchanged, so the model cannot predict the new ids.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.