Rank r in LoRA: what does the number actually count?

**Rank r** in LoRA is the integer that controls the size of the low-rank update matrices. It shows up in two specific places in the math, and it has two specific consequences. LoRA writes the weight update as a product of two skinny matrices: `delta_W = B * A`, where `B` has shape `d x r` and `A` has shape `r x k`. The `r` is the same number in both. It is the inner dimension where the two matrices meet. The two consequences fall out of the shapes. **First**, the total number of trainable parameters in the adapter is `r * (d + k)`, which grows linearly with r. A larger r means more parameters to train and store. **Second**, the rank of the product `BA` is at most r, which means the adapter can only move the weights in at most r independent directions of the high-dimensional weight space. Typical values: r between 4 and 8 works for narrow style adaptation. r between 16 and 64 is the common middle ground for broader behaviour changes. Above r = 128 you start approaching full-rank training costs without getting the full flexibility of true full fine-tuning.

**Rank r** is the shared inner dimension of the LoRA factorisation `delta_W = B * A`, with `B` of shape `d x r` and `A` of shape `r x k`. Two things flow directly from this definition. **The parameter count scales linearly with r.** Each adapted layer adds `r * (d + k)` trainable parameters, where d and k are the input and output dimensions of the original weight matrix. Doubling r roughly doubles the adapter size, the optimizer state, and the storage footprint. **The expressive ceiling is also r.** The rank of the product `BA` is at most r, so the adapter can move the underlying weights only along at most r linearly independent directions in weight space. If your task needs the model to change in more directions than r allows, no amount of training will close the gap. The practical regime breaks roughly into three: - `r = 4 to 8`: narrow style or domain adaptation, very cheap, often surprisingly effective for clear behavioural shifts. - `r = 16 to 64`: the common production middle ground for instruction tuning, persona shaping, and most fine-tunes published on Hugging Face. - `r >= 128`: approaches full-rank cost without offering the flexibility of true full fine-tuning, generally not worth it unless you have measured a quality gap that r=64 cannot close. LoRA's companion hyperparameter `alpha` scales the adapter contribution at inference time and is typically set to 2r or to r itself, depending on convention.