target_modules in a PEFT config: what does this list control?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

target_modules selects which submodules of the base transformer get wrapped with a LoRA adapter (i.e. which weight matrices receive a trainable low-rank update). The value is either a list of module-name suffixes, e.g. ['q_proj', 'v_proj'], or a regex, or the special string 'all-linear' to wrap every nn.Linear in the transformer blocks.

Classic LoRA targets only q_proj and v_proj (cheap, decent quality). 'all-linear' adds k_proj, o_proj, and the MLP projections (gate_proj, up_proj, down_proj), which raises trainable parameter count and typically lifts quality at higher cost. Modules NOT named here stay frozen at base weights.

Your answer

Quick reference

TL;DR

target_modules picks which linear submodules inside the transformer get a LoRA adapter wrapped around them. Everything not named stays frozen at base weights.

Memory aid

Easy to grasp

Think of the base model as a long row of doors. Behind each door is a giant grid of numbers the model uses for one specific job, like routing queries or expanding the hidden state. LoRA does not knock on every door. It only wraps adapters around the doors you point at. The target_modules argument is the list of door labels you hand over. Name two doors and only those two get an adapter; the rest stay locked and unchanged. Name all the doors and you wrap every one, which costs more compute but gives the adapter more places to learn. The list controls reach, and reach controls both quality and cost.

Key concepts

LoRA adapters Transformer architecture Parameter-efficient fine-tuning

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: define the argument, walk through the three forms it accepts, show the q-v versus all-linear tradeoff, and end with the cost and architecture-portability traps.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face PEFT exposes target_modules as a list, regex, or the special string 'all-linear' across every 2026 release.
Axolotl and Unsloth Llama 4 Maverick recipes typically set target_modules to 'all-linear' for code and math fine-tunes.
The QLoRA paper measured that targeting all linear layers consistently outperformed attention-only on GLUE and MMLU, and the recommendation has held in 2026 PEFT defaults.
Mistral 7B and Llama 4 share the same projection names, so a single target_modules list ports between them, while DeepSeek V4's MoE layers use mlp.experts.* paths that require regex matching.
Hugging Face PEFT documentation for Gemini 3.1 Pro and Gemma 4 includes architecture-specific target_modules examples because the projection names vary slightly across families.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the parameter count for a LoRA adapter scale with the choice of target_modules and the rank?

QWhy is all-linear typically the right default for hard domain shifts but overkill for style transfer?

QHow would you adapt target_modules for a multi-LoRA serving stack like vLLM with many adapters loaded at once?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Defaulting to q_proj and v_proj only on every project, regardless of difficulty. For complex domain shifts that under-targets the MLP path, where most of the model's capacity actually lives.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.