You want to add 5,000 Hindi medical terms to Llama 3 8B. Walk through the procedure and the risks.

Short answer·Medium·4.0 · 0·~3 min·Asked atComet Ml Mckinsey Tencent·Relevant atMeta Sarvam

Attempt it

Free · 2 AI evals / day

Adding tokens to a fine-tuned model has five steps and most teams skip at least one of them. **Step 1: choose the new tokens.** Train a small BPE on your Hindi medical corpus (5 to 10 GB), then take the top merges that are not already in Llama 3's 128K vocabulary. Cap the additions at your target (5,000). This is data-driven; hand-picking tokens is a worse choice because BPE is what the rest of the model expects. **Step 2: extend the tokenizer.** `tokenizer.add_tokens(new_tokens)`. This bumps the vocab size and assigns ids 128,256 through 133,255 to the new tokens. **Step 3: resize the model.** `model.resize_token_embeddings(len(tokenizer))`. This grows both the input embedding matrix and the lm_head to the new vocab size. HuggingFace resizes both together when they are tied. The new rows start as random. **Step 4: better initialization.** Random embeddings are gibberish until fine-tuning trains them. Better: initialize each new token's row as the mean of the embeddings of the subwords the token would have decomposed to under the old tokenizer. This gives the model a sensible starting point. **Step 5: fine-tune and save.** Fine-tune on enough Hindi medical data that the new embeddings learn meaningful representations. Save the tokenizer alongside the model with `tokenizer.save_pretrained` so inference loads the matching vocab. **Failure modes.** Catastrophic forgetting of English if you fine-tune Hindi-only; mix in 10 to 30 percent English. Under-trained additions produce gibberish at inference. Forgetting to save the tokenizer means inference loads the base 128K vocab and your new embedding rows are unreachable.

**Five-step procedure.** 1. **Select tokens via domain BPE.** Train a small BPE on 5 to 10 GB of Hindi medical text. Take the top merges not already in Llama 3's 128K vocabulary. Cap at 5,000. Data-driven selection beats hand-picking. 2. **Extend the tokenizer.** `tokenizer.add_tokens(new_tokens)`. New ids start at 128,256. 3. **Resize the model.** `model.resize_token_embeddings(len(tokenizer))` grows the input embedding matrix and the lm_head. HuggingFace handles both together when weights are tied; verify on your model class. 4. **Embedding-mean init.** Random init means the new tokens produce gibberish until trained. Better: for each new token, decompose it under the old tokenizer, take the embeddings of those subwords, average them. Same logic for lm_head rows. 5. **Fine-tune and save.** Under-trained additions still produce gibberish. After training, `tokenizer.save_pretrained(path)` alongside `model.save_pretrained(path)`. **Risks.** Catastrophic forgetting of English if Hindi-only; mix 10 to 30 percent English. Skipping the lm_head resize leaves the model unable to predict new ids. Mean-init fails for tokens with no clean subword decomposition; fall back to random for those. Forgetting the tokenizer save means inference uses the base 128K vocab and fine-tuned rows are unreachable. Vocab extension breaks some base instruction-tuned behavior the fine-tune has to compensate for.

TL;DR

Train BPE on Hindi, add_tokens, resize_token_embeddings (input AND lm_head), mean-init new rows, fine-tune with English mixed in, save the tokenizer. Watch forgetting and under-trained tokens.

Memory aid

Easy to grasp

Imagine teaching a friend who reads English to read Hindi medical reports. You decide to give them index cards for 5,000 common Hindi medical words so they do not have to spell each one letter by letter. The procedure has five parts. Pick the right 5,000 words by looking at lots of real Hindi medical text, not by guessing. Add the index cards to their card box and make sure their brain has space for the new words on both the 'reading' side and the 'speaking' side. Write a sensible first guess on each new card based on the spelling parts your friend already knows. Have them practice with enough Hindi medical text to actually learn the cards. Finally, give them the updated card box every time they go to read, not the old one. Skip any step and you get a friend who has forgotten some English, makes up nonsense for the new words, or cannot find the new cards at all.

Key concepts

Vocabulary Extension Fine-tuning Embedding Initialization

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Vocab extension is one of the higher-leverage fine-tune moves available when adapting an English-centric model to a new language or domain. Done right, it cuts fertility on the target distribution (fewer tokens per word means cheaper inference and longer effective context). Done wrong, it breaks the model in subtle ways that range from inability to generate the new tokens to catastrophic forgetting of the base capability.

The correct procedure has five steps: data-driven selection of the new tokens via a domain-trained BPE, extension of the tokenizer with add_tokens, resize of both the input embedding matrix and the lm_head, thoughtful initialization of the new rows (embedding-mean init rather than random), and sufficient fine-tuning to actually train the new rows. A sixth step that is often forgotten is saving the modified tokenizer alongside the model so inference loads the matching vocabulary.

The failure modes line up with the steps. Skip data-driven selection and you add tokens that do not match the domain. Skip the lm_head resize and the model cannot predict the new ids. Skip embedding-mean init and the model outputs gibberish for the new tokens until enough gradient flows through them. Skip data mixing and you catastrophically forget English. Skip the tokenizer save and the new rows are unreachable at inference. The rest of this explanation walks each layer in detail.

Step 1: choosing the new tokens

The temptation is to hand-pick 5,000 Hindi medical terms. The temptation is wrong. The rest of the model expects BPE-style sub-word structure, where common pieces are merged and rare pieces decompose into more common pieces. Hand-picked whole words break this property in awkward places.

The right approach is to train a small BPE on a Hindi medical corpus, around 5 to 10 GB. Sources include open Hindi medical journals, translated WHO content, and Hindi government health-ministry documentation. Train the BPE with a target vocabulary larger than what you want to add (say 20,000), then take the top merges that are not already in Llama 3's 128K vocabulary, and cap at 5,000. The top merges are by definition the highest-frequency Hindi patterns in your corpus, which is exactly what you want as additions.

A quality check at this step: tokenize a held-out Hindi medical sample with the proposed extended vocabulary and confirm fertility drops meaningfully versus the base. If the new tokens do not reduce fertility on your target distribution, the selection was wrong and the new tokens will not earn their parameter cost during inference.

Step 2 and 3: extending the tokenizer and resizing the model

Step 4: embedding-mean initialization

Step 5: fine-tuning, data mixing, and catastrophic forgetting

Step 6: persistence and inference compatibility

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tok = AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-8B')
model = AutoModelForCausalLM.from_pretrained('meta-llama/Meta-Llama-3-8B')

# 1. Select via domain BPE (sketch)
new_tokens = train_hindi_medical_bpe(corpus_path)[:5000]
new_tokens = [t for t in new_tokens if t not in tok.get_vocab()]

# 2. Add to tokenizer
num_added = tok.add_tokens(new_tokens)

# 3. Resize model (handles both input embedding and lm_head)
model.resize_token_embeddings(len(tok), pad_to_multiple_of=8)

# 4. Embedding-mean init for new rows
emb = model.get_input_embeddings().weight
lm_head = model.get_output_embeddings().weight
for i, tok_str in enumerate(new_tokens):
    sub_ids = tok.encode(tok_str, add_special_tokens=False)
    # Use the OLD subword decomposition for init
    new_id = len(tok) - num_added + i
    with torch.no_grad():
        emb[new_id] = emb[sub_ids].mean(dim=0)
        lm_head[new_id] = lm_head[sub_ids].mean(dim=0)

# 5. Fine-tune, then save tokenizer alongside model
model.save_pretrained(out_path)
tok.save_pretrained(out_path)  # critical

Real products, models, and research that use this idea.

OpenHathi (Sarvam AI) and Project Indus extended Llama 2 with Hindi tokens before fine-tuning on Hindi corpora; the published procedure uses BPE-trained additions plus mean init and explicitly warns about catastrophic forgetting.
HuggingFace transformers ships resize_token_embeddings with a pad_to_multiple_of parameter to keep new embedding matrices tensor-core aligned.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow many fine-tune tokens do you need before the new embedding rows are competitive with byte-fallback through the old tokenizer?

Rule of thumb: at least 100 occurrences of each new token across the fine-tune corpus, ideally several hundred. Below that, the new row never gets enough gradient signal to learn good representations and the model effectively still falls back to bytes internally. For 5,000 new tokens at 100 occurrences each, you need roughly 500K Hindi-token occurrences of the new tokens, which usually implies a few hundred MB of Hindi corpus minimum.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Calling add_tokens without resize_token_embeddings, or resizing only the input embedding while leaving the lm_head unchanged, so the model cannot predict the new ids.

60 second bullets to scan on the way to the call.

How to choose the 5,000 new tokens via a domain-trained BPE.
Why resize_token_embeddings must update both input embedding and lm_head.

Primary sources. Browse if you want the original framing.