target_modules in a PEFT config: what does this list control?
target_modules picks which linear submodules inside the transformer get a LoRA adapter wrapped around them. Everything not named stays frozen at base weights.
Think of the base model as a long row of doors. Behind each door is a giant grid of numbers the model uses for one specific job, like routing queries or expanding the hidden state. LoRA does not knock on every door. It only wraps adapters around the doors you point at. The target_modules argument is the list of door labels you hand over. Name two doors and only those two get an adapter; the rest stay locked and unchanged. Name all the doors and you wrap every one, which costs more compute but gives the adapter more places to learn. The list controls reach, and reach controls both quality and cost.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LoRA adapters do not change every weight in the base model. They wrap only the weight matrices you point them at, leaving everything else frozen. The target_modules argument in Hugging Face PEFT's LoraConfig is the mechanism for that pointing. It is one of the most consequential knobs in a LoRA recipe and one of the least often examined critically.
The question looks like a vocabulary check, but it opens onto real tradeoffs. The choice of which modules to target determines the trainable parameter count, the optimiser memory footprint, the wall-clock training time, and the achievable quality ceiling. Picking too narrowly under-trains; picking too widely wastes compute and can hurt convergence on small datasets.
This deep dive covers what the argument accepts, the standard presets, the cost and quality ladders, the architecture-portability traps, and the relationship between target_modules and the related modules_to_save argument.
What the argument accepts
The target_modules argument in PEFT's LoraConfig accepts three shapes.
The first is a list of name suffixes, like ['q_proj', 'v_proj']. PEFT walks the model's named modules and matches anything whose name ends with one of the listed suffixes. So 'model.layers.0.self_attn.q_proj' matches, as does 'model.layers.31.self_attn.q_proj', because both have the suffix q_proj. The list is short, readable, and the most common form in published recipes.
The second is a regex pattern, useful when the desired modules have a more complex naming scheme. Mixture-of-experts architectures like DeepSeek V4 nest expert weights under paths like mlp.experts.0.gate_proj, which a flat suffix list cannot disambiguate from a dense MLP. A regex like r'mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)' matches all expert MLPs and nothing else.
The third is the special string 'all-linear'. PEFT iterates every nn.Linear inside the transformer blocks and wraps it with a LoRA adapter. The lm_head and embedding layers are excluded by default because they are typically shared with input embeddings and full-training them requires the separate modules_to_save argument.
Each form does the same downstream thing: every matched module gets replaced by a LoraLinear wrapper that holds the frozen original weight plus the low-rank A and B factors. The wrapper's forward pass computes the base linear plus the scaled low-rank update.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face PEFT exposes target_modules as a list, regex, or the special string 'all-linear' across every 2026 release.
- Axolotl and Unsloth Llama 4 Maverick recipes typically set target_modules to 'all-linear' for code and math fine-tunes.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the parameter count for a LoRA adapter scale with the choice of target_modules and the rank?
For each wrapped linear of shape d_in by d_out, the adapter adds r times (d_in + d_out) parameters. Sum across wrapped layers to get the total; doubling rank or doubling the wrapped layer set both roughly double the count.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Defaulting to q_proj and v_proj only on every project, regardless of difficulty. For complex domain shifts that under-targets the MLP path, where most of the model's capacity actually lives.
60 second bullets to scan on the way to the call.
What target_modules selects in a transformer model
The three forms it accepts: list of names, regex, special string
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.