Compute the LoRA trainable param count for a 4096x4096 projection at rank r=8
B is 4096x8 and A is 8x4096, giving 65,536 trainable parameters per module, about 0.39 percent of the full 4096x4096 matrix.
Picture trying to repaint a giant wall. The wall is huge, so painting every single brick is expensive. Instead, you grab two narrow rollers. The first roller is tall but very thin, and the second is wide but very thin, and you press one against the other to make a pattern that paints the wall. The thin direction of each roller is the rank, a small number you pick. Because both rollers are skinny in that one direction, together they use a tiny fraction of the paint you would have spent doing every brick by hand. The final wall change still looks like a giant pattern, but the work you actually did sits in those two slim rollers.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Parameter count math is the second most asked LoRA question after what LoRA actually is. The reason is that the count drives every memory and cost number in the fine-tuning budget. Get the shapes wrong by a factor of two and your VRAM estimate is off by gigabytes; conflate the sum with the product and you are off by orders of magnitude.
The specific question here is concrete on purpose. A 4096 by 4096 linear layer is the shape of every attention projection in a Llama-3 8B or Mistral 7B, so this calculation is not academic. It is exactly the math you do when sizing a LoRA fine-tune in production. The answer drops out of one formula: trainable parameters equal rank times the sum of the input and output dimensions, and the resulting percentage is that number over the dense matrix size.
The rest of this section unpacks why that formula is the right one, why the parameter count sums rather than multiplies, why the initialisation of B matters for gradients to flow, and how the same calculation extends to multiple wrapped modules across a real model.
The B and A factorisation
LoRA adds a low-rank correction to a frozen dense weight. The correction is expressed as the product of two skinny matrices. If the frozen weight has shape d_out by d_in, the correction is also d_out by d_in, but it is built from B of shape d_out by r and A of shape r by d_in, where r is the rank.
Multiplying B times A reconstructs a matrix of the original shape. The forward pass becomes y equals W x plus B A x, where W is the frozen base and the product B A x is the LoRA contribution. The base weight is never touched. Only B and A are trainable.
For the 4096 by 4096 layer at r equals 8 in the question, B is 4096 by 8 and A is 8 by 4096. The product B A has the full 4096 by 4096 shape needed to add into the layer output, but the stored matrices are tiny. This is the structural reason LoRA saves memory: the reconstructed update has the original shape, but you only train and store the two skinny factors.
The rank r controls how expressive the correction can be. A rank-1 update is a single outer product, the lowest-capacity nonzero correction possible. A rank-r update is the sum of r outer products. As r climbs toward d_in or d_out, the low-rank constraint relaxes and the correction approaches the expressivity of a full dense update, at which point you have given up the parameter savings.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
# Wrap only the q_proj for one layer to verify the math
config = LoraConfig(
r=8,
lora_alpha=16,
target_modules=["q_proj"],
bias="none",
)
model = get_peft_model(base, config)
# For a 4096x4096 q_proj at r=8:
# B: 4096 x 8 = 32,768
# A: 8 x 4096 = 32,768
# total per wrapped module: 65,536
model.print_trainable_parameters()
# trainable params: ~32 * 65,536 (one per layer in a 32-layer base)
# trainable %: ~0.025% of the 8B base| rank r | params per 4096x4096 | % of full matrix |
|---|---|---|
| 8 | 65,536 | 0.39% |
| 16 | 131,072 | 0.78% |
| 32 | 262,144 | 1.56% |
| 64 | 524,288 | 3.13% |
| full FT | 16,777,216 | 100% |
Real products, models, and research that use this idea.
- Hugging Face PEFT LoraConfig at r=8 on a Llama 3.1 8B q_proj layer matches this exact 65,536 parameter count per wrapped module.
- Axolotl YAML configs default to r=16 alpha=32 on q and v projections for Mistral Large 3 fine-tunes, which is twice this calculation per layer.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is B initialised to zero rather than A?
If A were zero, the gradient of B would be zero times the upstream signal, and B would never move. Zeroing B keeps the adapter neutral at step zero while leaving A with a nonzero gradient through B's zero so updates begin immediately.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Computing B times A as 8x8 equals 64 instead of summing the two rectangles. The count is parameters in B PLUS parameters in A, not their product.
60 second bullets to scan on the way to the call.
The shapes of B and A as 4096 by r and r by 4096
Why parameter count is the sum, not the product, of the two matrices
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.