Compare LoRA r=8 vs r=64 on a 13B model on the quality vs cost curve

Short answer·Medium·4.0 · 0·~3 min·Asked atAutodesk Mphasis Yellow Ai·Relevant atCohere Coreweave Databricks Fireworks Ai

Attempt it

Sketch how the quality vs cost tradeoff actually looks when sweeping LoRA rank from r=8 to r=64 on a 13B model fine-tuned on a typical instruction dataset. Be concrete about how trainable parameters, wall clock training time, quality, and overfit risk move with rank.

Free · 2 AI evals / day

**LoRA rank** controls how expressive the adapter's update can be, and it sits at the centre of the LoRA cost-quality trade. Moving from r=8 to r=64 multiplies the LoRA parameter count by eight, because trainable parameters scale linearly with rank. But this does not multiply training time by eight. Most of the compute in a LoRA fine-tune is still the frozen base model's forward and backward pass, which does not change with rank. The adapter math is a small fraction of total compute, so wall-clock time grows only about 1.3 to 1.5x when you go from r=8 to r=64. Quality is where rank actually matters, and the curve is concave. r=8 often underfits on serious instruction work: the low-rank update is cramped, and capacity is the bottleneck. Bumping to r=16 or r=32 typically gives a clear quality lift. Going from r=32 to r=64 adds only a small benefit on most workloads, and on small datasets (under a few thousand examples) the extra capacity starts to memorise quirks rather than generalise. Adapter checkpoint size scales with parameter count, so r=64 adapters are eight times the disk space of r=8 adapters, but in absolute terms they are still megabytes, not gigabytes. The practical advice: start at r=16, push to r=32 if quality is short, only justify r=64 when the data is large enough and the distribution shift is broad enough to use it.

**Trainable parameters scale linearly with rank.** Moving from r=8 to r=64 multiplies LoRA params by 8x. On a 13B model with q, k, v, o projections wrapped across 40 layers (5120-dim attention), that is roughly 13 million parameters at r=8 and around 105 million at r=64. Wall-clock training time scales sub-linearly, growing about 1.3 to 1.5x over the same range. The reason: the frozen base forward and backward passes dominate compute. The adapter matmul on B times A times input is small. Memory delta is similarly modest because gradients and optimizer state grow with the adapter, not with the frozen base. Quality is the concave part of the curve. r=8 typically underfits on instruction work; capacity is the binding constraint, and outputs feel cramped. r=16 to r=32 is the sweet spot on most domains: enough capacity to express the task, not enough to memorise. r=64 buys little incremental quality on standard instruction datasets and starts to overfit on small ones (under a few thousand examples), where the extra capacity learns labeller idiosyncrasies rather than generalisable patterns. Adapter checkpoint size scales with the parameter count, so r=64 adapters are 8x larger on disk. In absolute terms this is megabytes to tens of megabytes, not a serving constraint. Decision rule: start at r=16, sweep to r=32 if eval reward is short, push to r=64 only when both dataset volume (10k or more examples) and distribution shift (a domain genuinely far from base) justify the extra capacity. Match alpha to r as a baseline so effective update magnitude stays constant.

TL;DR

Params scale 8x from r=8 to r=64, but wall time only grows ~1.4x. Quality plateaus around r=16 to 32; r=64 mostly helps when data is large or the shift is broad.

Memory aid

Easy to grasp

Imagine adding lanes to a highway. Going from one lane to two cuts traffic dramatically. Going from two to four still helps. But going from four to eight on a quiet road just adds asphalt nobody uses, and on a road with too few drivers, extra lanes invite people to weave around oddly and create chaos. Rank works the same way for a small training set. Adding capacity helps up to a point, then stops helping, and on a small road with low traffic, extra capacity starts learning the quirks of the specific drivers instead of generalising. The trick is finding the right number of lanes for the road you actually have.

Key concepts

LoRA adapters Rank selection Hyperparameter tuning

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Rank selection is the most common LoRA hyperparameter question in 2026 interviews and the most common quiet bug in production fine-tunes. The reason both is the same: the cost dimensions move at different rates, so a naive 'higher rank is more capable' intuition leads to a default of r=64 that is worse than r=32 on most workloads and worse than r=16 on small ones.

The specific comparison in this question, r=8 versus r=64 on a 13B model, is the right reference frame because those endpoints bracket the practical decision space. r=8 is where most teams start when memory is tight. r=64 is where many teams default when they want to play it safe. Both are usually wrong. The right answer almost always sits in the middle, but which middle value depends on the dataset and the distribution shift.

The rest of this section walks the four cost dimensions (parameters, wall-clock time, memory, adapter checkpoint size), the shape of the quality curve and where it plateaus, the overfit failure mode that appears at high rank on small data, and the decision rule that should drive rank selection in production.

The four cost dimensions and how they scale

Four cost dimensions matter for rank selection, and they each scale differently with r.

Trainable parameters scale linearly. For each wrapped module of shape d_out by d_in, the LoRA parameter count is r times (d_in plus d_out). Doubling r doubles the count exactly. On a 13B model wrapping the four attention projections per layer (q, k, v, o, each typically 5120 by 5120 on a 13B with 40 layers), r=8 lands around 13 million trainable parameters across the model (4 modules x 40 layers x 8 x 10,240); r=64 lands around 105 million. Both are tiny relative to the 13B base.

Wall-clock training time scales sub-linearly. Most compute is the frozen base forward and backward pass, which is independent of rank. The adapter matmul on B times A times input is small. Reported wall-time multipliers from r=8 to r=64 on H100-class hardware with bf16 and gradient checkpointing land at 1.3 to 1.5x. On consumer GPUs running QLoRA, the multiplier is slightly higher because dequantization overhead per adapter call adds up, but it is still much smaller than the parameter-count ratio would suggest.

Training memory grows modestly. Gradients and Adam moment buffers live on the adapter only, so they scale with adapter parameters, not base parameters. For r=8 the optimizer footprint is in the low single-digit gigabytes; for r=64 it grows but stays well under the activation footprint that dominates training memory. Sequence length and batch size move memory far more than rank does.

Adapter checkpoint size scales linearly with parameter count. r=8 adapters for a 13B model are typically 25 to 100 MB on disk; r=64 adapters are 200 MB to 800 MB. For single-tenant deployments this difference is irrelevant. For multi-tenant deployments hosting many adapters per frozen base, it shows up in storage cost and adapter load latency.

The shape of the quality curve

Overfit risk at high rank on small data

DoRA, alpha, and the production decision rule

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Axis	r=8	r=32	r=64
Trainable params (q,k,v,o on 13B)	~13M	~52M	~105M
Wall-clock (relative)	1.0x	~1.2x	~1.4x
Adapter checkpoint	~25 to 100 MB	~100 to 400 MB	~200 to 800 MB
Typical quality vs full FT	Underfits hard tasks	Close, often within 1 pt	Marginal lift over r=32
Overfit risk (small data)	Low	Moderate	High under 5k examples

Real products, models, and research that use this idea.

Hugging Face PEFT default tutorials use r=16 with alpha=32 for Llama 3.1 8B instruction fine-tunes on standard datasets.
Axolotl YAML configs in the community fine-tune ecosystem ship r=32 alpha=64 as a robust default for Mistral Large 3 single-GPU runs.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does wall-clock time scale sub-linearly with rank when parameter count scales linearly?

The flops budget per step is dominated by the frozen base forward and backward. The adapter matmul is a small fraction. Doubling rank doubles adapter compute but only marginally affects total step time. Gradient and optimizer state also live on the adapter only, so memory scales with the adapter, not the base.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Picking r=64 because higher rank sounds more capable. On most instruction datasets the gain over r=32 is invisible, and on small data the extra capacity actively hurts via overfitting.

60 second bullets to scan on the way to the call.

Why trainable params scale linearly with rank
Why wall-clock time scales sub-linearly because the frozen base dominates compute

Primary sources. Browse if you want the original framing.