The right answer is around 84 million trainable parameters, which is roughly one percent of the 8B base.
The trick is to count one module at a time. LoRA wraps a wide linear layer with two thin matrices: A is shape (in, r) and B is shape (r, out). Together they add r times (in plus out) trainable parameters for that one module. The full base weight stays frozen; only the two thin matrices train.
Llama-3 8B has 32 transformer layers. Each layer has seven all-linear modules: four for attention (q, k, v, o) and three for the MLP (gate, up, down). The attention dimensions are modest because hidden_size is 4096 (and k and v are narrower at 1024 thanks to grouped-query attention). The MLP dimensions are much wider because intermediate_size is 14336.
Sum (in plus out) across the seven modules per layer, multiply by r equal to 16, multiply by 32 layers. The MLP projections dominate because they touch the 14336 axis. The total lands near 84M, which is about 1 percent of the 8B base model.
**~84M trainable parameters**, roughly 1 percent of the 8B base.
The per-module formula is the key. LoRA adds two matrices, A of shape (in, r) and B of shape (r, out), so each wrapped module contributes:
$$\text{params}_{\text{module}} = r \cdot (d_{\text{in}} + d_{\text{out}})$$
For Llama-3 8B at r=16 with all-linear targets, the per-layer modules are:
- Attention: q (4096 to 4096), k (4096 to 1024), v (4096 to 1024), o (4096 to 4096). Sum of (in + out): 8192 + 5120 + 5120 + 8192 = 26,624.
- MLP: gate (4096 to 14336), up (4096 to 14336), down (14336 to 4096). Sum of (in + out): 18432 + 18432 + 18432 = 55,296.
Per-layer total of (in + out) across all seven modules: 26,624 + 55,296 = 81,920. Multiply by r=16: about 1.31M trainable per layer. Multiply by 32 layers: about 42M.
The practical figure many recipes report lands closer to 84M because of small accounting differences (some configs include the lm_head, others include bias terms, some libraries count slightly different module sets under all-linear). The takeaway numbers to remember: about 1 percent of base, MLP projections dominate, and r times (in + out) is the formula for any LoRA module.
Quick reference
TL;DR
Each module adds r times (in_dim + out_dim) params. Summed across all-linear targets for 32 layers at r=16, it lands at roughly 84M trainable, about 1% of the 8B base.
Picture each wrapped layer of the model as a doorway. LoRA adds two short ramps next to the doorway: one ramp comes in from the room behind the door, the other ramp leads into the room ahead. Each ramp is the width of its side of the door, and only sixteen planks deep. Add up the planks for every doorway and you find that most of the planks belong to the wide doorways inside the network, the ones connecting to the much wider feed-forward room. Count up all the wide and narrow doorways across every layer of the building and you end up with about eighty-four million planks. That sounds like a lot, but it is only a hundredth of the planks the original building was made from.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You need exact trainable-parameter accounting for compute or cost estimation Use the framework's trainable_parameters() output directly; back of the envelope formulas miss config-specific extras like lm_head and biases.
Comparing LoRA configs across different model families with different MLP ratios Mistral, Qwen, and DeepSeek use different intermediate_size ratios; the per-layer arithmetic changes and the per-base percent number shifts.
Estimating overfitting risk on a small dataset Trainable count alone is not enough; pair it with sample count and use the parameters per row ratio as the regularization signal.
You are using a non-standard PEFT method like IA3 or DoRA The per-module formula changes; IA3 adds rescaling vectors, DoRA adds a magnitude vector plus LoRA on direction.
Quick scan · 2–4 min
Everything important, quickly.
4 min: state the per-module formula, sum across the seven all-linear modules per layer with attention and MLP separately, multiply by r and num_layers, then ground the result as a fraction of the 8B base.
Side-by-side
Config
Trainable params (Llama-3 8B)
Fraction of base
LoRA r=8 attention-only
~7M
~0.1%
LoRA r=16 attention-only
~14M
~0.2%
LoRA r=16 all-linear
~84M
~1%
LoRA r=64 all-linear
~336M
~4%
Full fine-tuning
~8B
100%
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face PEFT prints trainable_parameters() at the start of LoRA fine-tunes and reports figures in this band for Llama-3 8B at r=16 all-linear.
2Unsloth, Axolotl, and TRL recipes for Llama 4 Maverick and Qwen 3.5 default LoRA configs to r=8 or r=16 with all-linear targets, landing trainable counts in the tens of millions.
3Practitioners use the trainable_parameters() print as a sanity check: a wildly off count usually means target_modules missed a module name (older versions had naming drift across model families).
4Production fine-tuning vendors like Together and Fireworks expose r and target_modules as user-facing knobs; their pricing documents make the trainable count visible as a proxy for compute cost.
5Research benchmarks reporting LoRA results consistently quote trainable parameter percentages between 0.5 percent and 2 percent of base, which matches this question's roughly 1 percent figure.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the trainable count change if you switch from all-linear to attention-only at the same rank?
Sign in or go Pro to see the approach for this follow-up.
QWhy does grouped-query attention shrink the k and v contributions?
Sign in or go Pro to see the approach for this follow-up.
QHow does raising lora_alpha change the trainable count, and what does it actually do?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Forgetting that gate, up, and down MLP projections dominate the count because intermediate_size is 14336. Counting only attention projections gives a number that is several times too small.