**modules_to_save** is a PEFT config field that lists modules to train **fully**, in full precision and full rank, alongside the LoRA adapters that wrap other modules. The modules in this list bypass the LoRA decomposition entirely; their original weights are unfrozen and updated by the optimizer as if you were doing full fine-tuning on just those modules.
The canonical use case is **tokenizer extension**. You take a base model and add new tokens to its vocabulary, for example domain-specific markers like `<|FUNCTION|>` or new language script characters that the base tokenizer never saw. Adding new tokens grows two specific weight tensors:
- `embed_tokens`, the input embedding lookup table. The new tokens get fresh rows initialised randomly.
- `lm_head`, the output projection that maps hidden states back to vocab logits. New tokens get fresh rows there too.
Those fresh rows start at random. A LoRA adapter wrapping these layers cannot effectively train the new rows because the low-rank update is added to the original weight, and the original weight at the new rows is just random noise. Training those modules fully (with modules_to_save) lets the new rows learn proper representations.
The other options describe things modules_to_save does not control. Rank, sharding, and adapter loading are governed by other config fields entirely.
**modules_to_save** marks specific modules to train fully in full rank alongside the LoRA adapter wrappers on everything else. The canonical use case is vocabulary extension.
When you extend the tokenizer with new tokens (say, for a new domain, a new language script, or special framing markers), two weight tensors gain new rows:
- `embed_tokens` of shape `[vocab_size, d_model]`: new rows for the new tokens, initialised randomly.
- `lm_head` of shape `[vocab_size, d_model]`: new rows for predicting those tokens, also random.
Those rows have to learn from scratch. A LoRA adapter wrapping the embedding layer would add a low-rank delta `(alpha / r) * B A` to the original weight, but the original at the new-token rows is random noise. The low-rank delta cannot rescue noisy initialization; you would be trying to make sense of random plus a low-rank correction.
The fix is to list both modules in modules_to_save:
```
LoraConfig(
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
modules_to_save=["embed_tokens", "lm_head"]
)
```
Now PEFT unfreezes embed_tokens and lm_head, leaves them out of the LoRA wrapping, and trains them in full precision and full rank. The new token rows learn proper representations from the data, and the existing rows can refine if the optimizer thinks they should.
The other options describe unrelated knobs. Target modules are listed in target_modules. Sharding is FSDP or DeepSpeed config. Adapter loading is controlled by load_adapter and merge methods.
Quick reference
TL;DR
modules_to_save trains listed modules fully alongside the LoRA adapter, typically embed_tokens and lm_head when the tokenizer gains new tokens whose random rows LoRA cannot reach.
Imagine your model is a giant library with a fixed catalogue of book topics. LoRA is a small set of margin notes you can scribble on existing pages to adjust how they read. Then one day you add brand new book categories to the library, categories the original catalogue never had. Those new categories need fresh full pages in the index, not margin notes. modules_to_save is the list of pages you mark as needing fresh full pages instead of margin notes. The most common case is adding new words to the model's vocabulary; the input lookup table and the output prediction layer both need new full pages for the new words to make sense.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Standard LoRA fine-tuning with no tokenizer changes and no new architecture There is nothing that needs full rank training; LoRA on the target modules covers all the learning needed, and modules_to_save is unnecessary overhead.
Memory-constrained QLoRA runs where the embedding and output projection take significant memory Adding these modules to modules_to_save undoes the memory benefit of quantisation and may exceed the GPU budget.
Fine-tuning where the goal is purely behavior change, not capability extension Behavior change is what LoRA was designed for; modules_to_save is for capability changes that require new rows in the embedding or output.
Models where embed_tokens and lm_head are tied and you only want to update some of the new rows Tied weights propagate updates to both views simultaneously; partial updates require explicit weight untying or a different fine-tuning strategy.
Quick scan · 2–4 min
Everything important, quickly.
4 min: name modules_to_save's semantics, place vocab extension as the canonical use case, list embed_tokens and lm_head, contrast with target_modules, and surface the tied weight and memory caveats.
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face PEFT LoraConfig exposes modules_to_save as a top-level field used in Llama 4 fine-tuning recipes when adding domain markers to the tokenizer.
2Axolotl recipes for multilingual continued training of Mistral list embed_tokens and lm_head in modules_to_save when extending the tokenizer for new scripts.
3DeepSeek V4 distillation pipelines that add special function-calling tokens use modules_to_save to ensure the embedding and output projection rows for the new tokens train fully.
4Unsloth tutorials for fine-tuning Gemma 4 on tool-use data demonstrate modules_to_save with embed_tokens and lm_head when introducing new tool framing tokens.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is LoRA fundamentally unable to teach new vocabulary tokens added after pretraining?
Sign in or go Pro to see the approach for this follow-up.
QWhen tying embed_tokens and lm_head, how does modules_to_save behave?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the memory cost of adding embed_tokens to modules_to_save for a model with a large vocabulary?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating modules_to_save as a list of LoRA target modules. It is the opposite: these modules bypass LoRA entirely and train at full precision in full rank.