Compare LoRA r=8 vs r=64 on a 13B model on the quality vs cost curve
Sketch how the quality vs cost tradeoff looks when sweeping LoRA rank from r=8 to r=64 on a 13B model fine-tuned on a typical instruction dataset. Be concrete about how trainable parameters, wall clock training time, quality, and overfit risk move with rank.
Params scale 8x from r=8 to r=64, but compute grows only about 1%. Quality plateaus around r=16 to 32. r=64 mostly helps when data is large or the shift is broad.
Imagine adding lanes to a highway. Going from one lane to two cuts traffic a lot. Going from two to four still helps. But going from four to eight on a quiet road just adds asphalt nobody uses. On a road with too few drivers, extra lanes invite people to weave around oddly and create chaos. Rank works the same way for a small training set. Adding capacity helps up to a point, then stops helping. On a small road with low traffic, extra capacity starts learning the quirks of the specific drivers instead of generalising. The trick is finding the right number of lanes for the road you have.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Rank selection is a common quiet mistake in production LoRA fine-tunes. The reason is that the cost dimensions move at different rates. A naive 'higher rank is more capable' intuition leads to a default of r=64. That default is no better than r=32 on most workloads and worse than r=16 on small ones.
On a 13B model, r=8 and r=64 bracket the practical decision space. r=8 is where most teams start when memory is tight. r=64 is where many teams default when they want to play it safe. Both are usually wrong. The right rank almost always sits in the middle. Which middle value depends on the dataset and the distribution shift.
The four cost dimensions and how they scale
Four cost dimensions matter for rank selection, and they each scale differently with r.
Trainable parameters scale linearly. For each wrapped module of shape d_out by d_in, the LoRA parameter count is r times (d_in plus d_out). Doubling r doubles the count exactly. On a 13B model wrapping the four attention projections per layer (q, k, v, o, each typically 5120 by 5120 on a 13B with 40 layers), r=8 lands around 13 million trainable parameters across the model (4 modules x 40 layers x 8 x 10,240). r=64 lands around 105 million. Both are tiny relative to the 13B base.
Wall-clock training time scales sub-linearly. Most compute is the frozen base forward and backward pass, which is independent of rank. The adapter matmul on B times A times input is small. Counted in FLOPs, the adapter adds about 0.2% to a training step at r=8 and about 1.2% at r=64 on Llama-2-13B shapes. Measured step time rises somewhat more, because the small adapter matmuls run less efficiently than the large base ones. On consumer GPUs running QLoRA, dequantizing the base adds a cost per step that does not grow with rank. The multiplier is still much smaller than the parameter count ratio would suggest.
Training memory grows modestly. Gradients and Adam moment buffers live on the adapter only, so they scale with adapter parameters, not base parameters. For r=8 the optimizer footprint is under a gigabyte. For r=64 it grows but stays well under the activation footprint that dominates training memory. Sequence length and batch size move memory far more than rank does.
Adapter checkpoint size scales linearly with parameter count. With the four attention projections wrapped, r=8 adapters for a 13B model are about 26 MB on disk in bf16 (52 MB in fp32). r=64 adapters are about 210 MB (420 MB in fp32). For single-tenant deployments this difference is irrelevant. For multi-tenant deployments hosting many adapters per frozen base, it shows up in storage cost and adapter load latency.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Axis | r=8 | r=32 | r=64 |
|---|---|---|---|
| Trainable params (q,k,v,o on 13B) | ~13M | ~52M | ~105M |
| Adapter share of training FLOPs | ~0.2% | ~0.6% | ~1.2% |
| Adapter checkpoint (bf16 to fp32) | ~26 to 52 MB | ~105 to 210 MB | ~210 to 420 MB |
| Typical quality vs full FT | Underfits hard tasks | Close on near-base tasks | Marginal lift over r=32 |
| Overfit risk (small data) | Low | Moderate | High under 5k examples |
Real products, models, and research that use this idea.
- Many community LoRA recipes for 7B to 8B instruction fine-tunes start at r=16 with alpha=32.
- Axolotl's example LoRA and QLoRA configs for Llama and Mistral models use r=32 with lora_alpha=16.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does wall-clock time scale sub-linearly with rank when parameter count scales linearly?
The frozen base forward and backward dominate the flops budget per step. The adapter matmul is a small fraction. Doubling rank doubles adapter compute but only marginally affects total step time. Gradient and optimizer state also live on the adapter only, so memory scales with the adapter, not the base.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking r=64 because higher rank sounds more capable. On most instruction datasets the gain over r=32 is invisible, and on small data the extra capacity actively hurts via overfitting.
60 second bullets to scan on the way to the call.
Why trainable params scale linearly with rank
Why wall-clock time scales sub-linearly because the frozen base dominates compute
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.