Rank r in LoRA: what does the number actually count?

Flashcard·Easy·4.0 · 0·~30s·Asked atSambanova Sap Weaviate·Relevant atDatabricks

Attempt it

r is the shared inner dimension of the low-rank factorisation ΔW = B · A: B has shape d × r and A has shape r × k. So r controls (a) how many trainable parameters the adapter has, it grows linearly with r, and (b) the maximum rank of the update matrix BA, which caps how many independent directions in weight space the adapter can move. Tiny r (4 to 8) is fine for narrow style adaptation; broader behaviour changes typically want r in the 16 to 64 range. Above ~128, returns diminish and you start approaching full-rank costs without the full-rank flexibility.

Your answer

**Rank r** in LoRA is the integer that controls the size of the low-rank update matrices. It shows up in two specific places in the math, and it has two specific consequences. LoRA writes the weight update as a product of two skinny matrices: `delta_W = B * A`, where `B` has shape `d x r` and `A` has shape `r x k`. The `r` is the same number in both. It is the inner dimension where the two matrices meet. The two consequences fall out of the shapes. **First**, the total number of trainable parameters in the adapter is `r * (d + k)`, which grows linearly with r. A larger r means more parameters to train and store. **Second**, the rank of the product `BA` is at most r, which means the adapter can only move the weights in at most r independent directions of the high-dimensional weight space. Typical values: r between 4 and 8 works for narrow style adaptation. r between 16 and 64 is the common middle ground for broader behaviour changes. Above r = 128 you start approaching full-rank training costs without getting the full flexibility of true full fine-tuning.

**Rank r** is the shared inner dimension of the LoRA factorisation `delta_W = B * A`, with `B` of shape `d x r` and `A` of shape `r x k`. Two things flow directly from this definition. **The parameter count scales linearly with r.** Each adapted layer adds `r * (d + k)` trainable parameters, where d and k are the input and output dimensions of the original weight matrix. Doubling r roughly doubles the adapter size, the optimizer state, and the storage footprint. **The expressive ceiling is also r.** The rank of the product `BA` is at most r, so the adapter can move the underlying weights only along at most r linearly independent directions in weight space. If your task needs the model to change in more directions than r allows, no amount of training will close the gap. The practical regime breaks roughly into three: - `r = 4 to 8`: narrow style or domain adaptation, very cheap, often surprisingly effective for clear behavioural shifts. - `r = 16 to 64`: the common production middle ground for instruction tuning, persona shaping, and most fine-tunes published on Hugging Face. - `r >= 128`: approaches full-rank cost without offering the flexibility of true full fine-tuning, generally not worth it unless you have measured a quality gap that r=64 cannot close. LoRA's companion hyperparameter `alpha` scales the adapter contribution at inference time and is typically set to 2r or to r itself, depending on convention.

TL;DR

Rank r is the shared inner dimension of the LoRA factorisation. It linearly controls trainable parameter count and caps how many directions in weight space the adapter can move.

Memory aid

Easy to grasp

Picture trying to summarise a thick novel using a fixed number of index cards. With four cards you can capture the broadest strokes; with sixty-four you can keep most of the nuance; with two hundred you are essentially copying the book. The number of cards is the rank. In LoRA, rank r decides how many independent directions of change the adapter has to work with. Too few cards and you cannot capture the new behaviour. Too many and you have lost the whole point of a compact summary. The sweet spot sits in the middle, big enough to express what you need, small enough to stay cheap. The smaller the number, the more compressed the change has to be.

Key concepts

LoRA PEFT Low-rank factorisation LoRA alpha scaling

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Rank r is the most important hyperparameter in LoRA and one of the most commonly misunderstood. People know it controls some kind of capacity, but few can explain precisely where it appears in the math, what it costs in parameters and memory, or how it interacts with the companion hyperparameter alpha. Getting this wrong leads to fine-tunes that quietly underfit, fine-tunes that overfit small datasets, or fine-tunes that throw away LoRA's storage benefits by picking r so high you are nearly back to full-rank cost.

This deep dive walks through the geometry of LoRA, locates where r appears in the factorisation, derives the parameter count and the expressive ceiling from the shapes, explains the alpha/r scaling that turns r into a magnitude knob as well as a capacity knob, and surveys the typical r ranges that have emerged across the open-weight ecosystem.

The headline is that r controls two things at once. Trainable parameter count grows linearly with r because the adapter matrices have shape d x r and r x k. The expressive ceiling is also r, because the rank of the update matrix BA cannot exceed the shared inner dimension. These two consequences are tightly coupled, and tuning r means navigating the trade-off between expressive capacity and the cost (in parameters, optimizer state, and serving memory) of providing that capacity.

For most production work, r lives between 8 and 64. Outside that range you should have a specific reason: either the task is narrow enough that r=4 is sufficient, or you have measured a quality gap that genuinely requires r above 64. Default to r=8 or r=16 for new fine-tunes and tune from there based on measured behaviour, not based on the intuition that bigger is better.

Where r appears in the LoRA factorisation

The factorisation

LoRA replaces a full-rank weight update with a low-rank one. If the original weight matrix is W of shape d x k, the update is written as:

\Delta W = B A

where B has shape d x r and A has shape r x k. The full updated weight at inference time is W + (alpha / r) * B A. During training, only B and A are updated; W stays frozen.

The two roles of r

The integer r plays two roles simultaneously. First, it is the second dimension of B and the first dimension of A, so it controls how many trainable parameters live in the adapter. The total parameter count per adapted layer is:

r \cdot (d + k)

Second, it is the rank of the product matrix BA. By the rank-of-product inequality, the rank of BA is at most min(d, r, k), which for typical large layers (r much smaller than d and k) equals r. So r is also the maximum number of linearly independent directions in weight space that the adapter can move.

Initialisation matters

B is initialised to zero. A is initialised with small random Gaussian values. This is deliberate: at step zero the product BA is the zero matrix, so the adapted model behaves identically to the base. Once gradients start flowing, B lifts off zero and the adapter begins to deviate. Symmetric zero initialisation on both matrices would freeze the adapter at zero forever because the gradient through a zero product is zero in both directions.

Parameter count, memory, and the linear scaling

The expressive ceiling and the intrinsic-rank hypothesis

Alpha, the alpha/r scaling, and picking r in practice

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Rank r	Trainable params (per 4096x4096 layer)	Typical use case
4	~32k	Narrow style adaptation
8	~65k	Default starting point
16	~131k	Common instruction tuning
64	~524k	Broad behaviour shaping
128	~1M	Diminishing returns territory

Real products, models, and research that use this idea.

Hugging Face PEFT defaults LoRA r to 8 in its example configs, a starting point most teams keep for initial experiments.
QLoRA fine-tunes of Llama 4 Maverick commonly use r=64 with alpha=16 on all linear layers, balancing capacity against single-GPU memory.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does LoRA initialise B to zero and A randomly?

Zero-init on B ensures the adapter starts as the identity (the product BA is zero), so the model behaves exactly like the base at step zero. Random A ensures gradients can flow through both matrices once training starts. Symmetric zero-init would freeze the adapter at zero forever.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing rank r with the number of LoRA layers. r is the inner dimension of each adapter matrix; the number of layers and which modules get adapted are separate config choices.

60 second bullets to scan on the way to the call.

Where r appears in the LoRA factorisation
The formula for adapter parameters per layer

Primary sources. Browse if you want the original framing.