When does a PEFT config actually need `modules_to_save` set, and what goes in it?

MCQ·Easy·4.0 · 0·~1 min

Attempt it

**modules_to_save** is a PEFT config field that lists modules to train **fully**, in full precision and full rank, alongside the LoRA adapters that wrap other modules. The modules in this list bypass the LoRA decomposition entirely; their original weights are unfrozen and updated by the optimizer as if you were doing full fine-tuning on just those modules. The canonical use case is **tokenizer extension**. You take a base model and add new tokens to its vocabulary, for example domain-specific markers like `<|FUNCTION|>` or new language script characters that the base tokenizer never saw. Adding new tokens grows two specific weight tensors: - `embed_tokens`, the input embedding lookup table. The new tokens get fresh rows initialised randomly. - `lm_head`, the output projection that maps hidden states back to vocab logits. New tokens get fresh rows there too. Those fresh rows start at random. A LoRA adapter wrapping these layers cannot effectively train the new rows because the low-rank update is added to the original weight, and the original weight at the new rows is just random noise. Training those modules fully (with modules_to_save) lets the new rows learn proper representations. The other options describe things modules_to_save does not control. Rank, sharding, and adapter loading are governed by other config fields entirely.

**modules_to_save** marks specific modules to train fully in full rank alongside the LoRA adapter wrappers on everything else. The canonical use case is vocabulary extension. When you extend the tokenizer with new tokens (say, for a new domain, a new language script, or special framing markers), two weight tensors gain new rows: - `embed_tokens` of shape `[vocab_size, d_model]`: new rows for the new tokens, initialised randomly. - `lm_head` of shape `[vocab_size, d_model]`: new rows for predicting those tokens, also random. Those rows have to learn from scratch. A LoRA adapter wrapping the embedding layer would add a low-rank delta `(alpha / r) * B A` to the original weight, but the original at the new-token rows is random noise. The low-rank delta cannot rescue noisy initialization; you would be trying to make sense of random plus a low-rank correction. The fix is to list both modules in modules_to_save: ``` LoraConfig( target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], modules_to_save=["embed_tokens", "lm_head"] ) ``` Now PEFT unfreezes embed_tokens and lm_head, leaves them out of the LoRA wrapping, and trains them in full precision and full rank. The new token rows learn proper representations from the data, and the existing rows can refine if the optimizer thinks they should. The other options describe unrelated knobs. Target modules are listed in target_modules. Sharding is FSDP or DeepSpeed config. Adapter loading is controlled by load_adapter and merge methods.

Quick reference

TL;DR

modules_to_save trains listed modules fully alongside the LoRA adapter, typically embed_tokens and lm_head when the tokenizer gains new tokens whose random rows LoRA cannot reach.

Memory aid

Easy to grasp

Imagine your model is a giant library with a fixed catalogue of book topics. LoRA is a small set of margin notes you can scribble on existing pages to adjust how they read. Then one day you add brand new book categories to the library, categories the original catalogue never had. Those new categories need fresh full pages in the index, not margin notes. modules_to_save is the list of pages you mark as needing fresh full pages instead of margin notes. The most common case is adding new words to the model's vocabulary; the input lookup table and the output prediction layer both need new full pages for the new words to make sense.

Key concepts

modules_to_save config Parameter-efficient fine-tuning Tokenizer extension LoRA adapters

Detailed answer & concept explanation

~10 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: name modules_to_save's semantics, place vocab extension as the canonical use case, list embed_tokens and lm_head, contrast with target_modules, and surface the tied weight and memory caveats.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face PEFT LoraConfig exposes modules_to_save as a top-level field used in Llama 4 fine-tuning recipes when adding domain markers to the tokenizer.
Axolotl recipes for multilingual continued training of Mistral list embed_tokens and lm_head in modules_to_save when extending the tokenizer for new scripts.
DeepSeek V4 distillation pipelines that add special function-calling tokens use modules_to_save to ensure the embedding and output projection rows for the new tokens train fully.
Unsloth tutorials for fine-tuning Gemma 4 on tool-use data demonstrate modules_to_save with embed_tokens and lm_head when introducing new tool framing tokens.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is LoRA fundamentally unable to teach new vocabulary tokens added after pretraining?

QWhen tying embed_tokens and lm_head, how does modules_to_save behave?

QWhat is the memory cost of adding embed_tokens to modules_to_save for a model with a large vocabulary?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating modules_to_save as a list of LoRA target modules. It is the opposite: these modules bypass LoRA entirely and train at full precision in full rank.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.