Estimate the trainable param count for a Llama-3 8B LoRA at r=16 targeting all linear modules.
from peft import LoraConfig import torch # Base: meta-llama/Meta-Llama-3-8B # hidden_size = 4096 # intermediate_size = 14336 # num_layers = 32 # num_kv_heads = 8 (head_dim 128 -> kv_proj out = 1024) # # LoraConfig( # r=16, lora_alpha=32, bias="none", # target_modules="all linear", # q,k,v,o + gate,up,down per layer # ) # # Report the order of magnitude trainable parameter count # (one significant figure is enough).
Each module adds r times (in_dim + out_dim) params. Summed across all-linear targets for 32 layers at r=16, it lands at roughly 84M trainable, about 1% of the 8B base.
Picture each wrapped layer of the model as a doorway. LoRA adds two short ramps next to the doorway: one ramp comes in from the room behind the door, the other ramp leads into the room ahead. Each ramp is the width of its side of the door, and only sixteen planks deep. Add up the planks for every doorway and you find that most of the planks belong to the wide doorways inside the network, the ones connecting to the much wider feed-forward room. Count up all the wide and narrow doorways across every layer of the building and you end up with about eighty-four million planks. That sounds like a lot, but it is only a hundredth of the planks the original building was made from.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Parameter counting for LoRA looks like arithmetic and is really a question about which modules dominate the trainable surface. The answer for Llama-3 8B at r=16 with all-linear targeting is about 84M, roughly 1 percent of the 8B base. The path to that answer reveals the structure of modern decoder LLMs.
The formula is short. Each LoRA module adds two thin matrices, A of shape (in, r) and B of shape (r, out), for a total of r times (in plus out) trainable parameters per module. The full base weight stays frozen. To estimate the count for a configuration, sum (in + out) across every targeted module, multiply by r, and you are done.
The interesting part is which modules contribute most. On Llama-3 8B, the MLP projections (gate, up, down) operate on the intermediate_size of 14336, much wider than the hidden_size of 4096. They dominate the per-layer parameter count. Attention projections benefit from grouped-query attention shrinking the k and v outputs, which makes them noticeably cheaper.
This deep dive walks through the per-module formula, sums it across the seven all-linear targets per layer, multiplies by 32 layers, grounds the answer as a fraction of base, and explains why the reported number can drift slightly from the back of the envelope calculation.
The per-module formula
Start from what LoRA actually trains. The base model has a weight matrix W of shape (out, in) at every linear layer. LoRA freezes W and adds a low-rank update BA, where A has shape (in, r) and B has shape (r, out). At forward time the effective weight is W plus BA scaled by alpha over r.
The trainable parameters per module are the entries of A and B. A has in times r entries, B has r times out entries. The sum is r times (in plus out). That is the only formula you need for any single LoRA module, regardless of which linear layer it wraps.
A quick sanity check on the math. If the base module has 16 million parameters (in equal to out equal to 4096), and you wrap it with r=16, the LoRA contribution is 16 times (4096 + 4096), which is 131,072 trainable parameters. That is less than 1 percent of the base module's parameter count, which is exactly why LoRA is called parameter-efficient. The full module stays frozen; only the thin matrices train.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Config | Trainable params (Llama-3 8B) | Fraction of base |
|---|---|---|
| LoRA r=8 attention-only | ~7M | ~0.1% |
| LoRA r=16 attention-only | ~14M | ~0.2% |
| LoRA r=16 all-linear | ~84M | ~1% |
| LoRA r=64 all-linear | ~336M | ~4% |
| Full fine-tuning | ~8B | 100% |
Real products, models, and research that use this idea.
- Hugging Face PEFT prints trainable_parameters() at the start of LoRA fine-tunes and reports figures in this band for Llama-3 8B at r=16 all-linear.
- Unsloth, Axolotl, and TRL recipes for Llama 4 Maverick and Qwen 3.5 default LoRA configs to r=8 or r=16 with all-linear targets, landing trainable counts in the tens of millions.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the trainable count change if you switch from all-linear to attention-only at the same rank?
Drop the MLP contributions from the sum. On Llama-3 8B that removes about two thirds of the per-layer (in + out) sum, so the trainable count falls by roughly 3x. The cost is reduced expressivity, especially for tasks that need MLP changes (knowledge injection, format learning).
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that gate, up, and down MLP projections dominate the count because intermediate_size is 14336. Counting only attention projections gives a number that is several times too small.
60 second bullets to scan on the way to the call.
The per-module LoRA formula in terms of r, in_dim, and out_dim
Why MLP projections dominate the count on modern decoder LLMs
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.