Compute the LoRA trainable param count for a 4096x4096 projection at rank r=8

**LoRA** wraps a frozen linear layer with a small, trainable add-on. Instead of touching the original weight matrix, it learns a tiny correction expressed as the product of two skinny matrices. The full layer here is 4096 by 4096, which contains 4096 times 4096 equals about 16.8 million parameters. LoRA does not retrain any of those. It adds a side path made of two pieces called A and B, joined at a small inner dimension called the rank. At rank 8 the shapes are: B is 4096 by 8, and A is 8 by 4096. Multiply them together and the product is a 4096 by 4096 matrix that gets added to the original. Crucially, the only things being trained are the entries of B and A themselves. Count them. B has 4096 times 8 equals 32,768 entries. A has 8 times 4096 equals another 32,768 entries. Add them: 65,536 trainable parameters per wrapped layer. As a fraction of the full matrix, 65,536 divided by 16,777,216 is about 0.39 percent. So LoRA at rank 8 trains less than half of one percent of what full fine-tuning would touch.

**LoRA** factors its update into two low-rank matrices. The wrapped layer is 4096 by 4096. At rank r equals 8, the adapter shapes are: - B has shape 4096 by 8 - A has shape 8 by 4096 The trainable parameter count is the sum of entries in both: $$\text{params} = r \cdot (d_{\text{in}} + d_{\text{out}}) = 8 \cdot (4096 + 4096) = 65{,}536$$ That is 65,536 trainable parameters per wrapped module. Compare to the full matrix size of 4096 times 4096 equals 16,777,216, and the ratio is 65,536 divided by 16,777,216 equals about 0.39 percent. The forward pass becomes y equals W x plus B times A times x, where W is the frozen pretrained weight and the product B A produces the correction. The rank r controls capacity: higher r adds more parameters linearly but does not multiply the storage cost the way a full retrain would. A LoRA at rank 16 on the same layer is 131,072 parameters; rank 64 is 524,288. Even at r equals 64 you sit under 4 percent of the full layer count. Most adapters wrap only the attention projections (q, k, v, o) rather than every linear layer, so a 13B model fine-tune typically lands between 10 and 100 million trainable parameters total.