LoRA freezes the base weights: does that mean activation checkpointing on the base is unnecessary?

MCQ·Medium·4.0 · 0·~1 min·Asked atSarvam Vernacular Ai Writer·Relevant atCohere Databricks Meta Microsoft

Attempt it

The correct answer is **A**. Freezing the base weights does not remove activations from the picture, and **gradient checkpointing** on those base modules still pays off. Here is the core distinction. Freezing a weight means the optimizer never updates it, so no gradient needs to be computed for that weight specifically. That saves the gradient buffer on those weights. But the chain rule still requires the activations of each layer to compute gradients for anything downstream that does need them, and that includes the LoRA adapters sitting on top. When the loss gradient flows backward, it passes through every base-layer operation that produced the activation feeding the LoRA adapter. Each of those operations has a local Jacobian that depends on its input. Without the cached input activation, the Jacobian cannot be evaluated, so the gradient cannot continue backward into the adapter. The other options are wrong for specific reasons. B confuses no gradient on weight with no activations needed. C invents a pre-norm versus post-norm distinction that the chain rule does not respect. D invents an optimizer-specific rule, but activation memory is decided by the autograd graph, not by which optimizer reads the gradients afterward. The practical takeaway: LoRA runs on long contexts still hit activation memory walls, and checkpointing on the base modules is one of the most useful levers for fitting them.

**The answer is A**, and the right framing is to separate two distinct concepts that the trap option B fuses incorrectly. Freezing a weight removes gradient computation for that weight. It does not remove the requirement to cache the activations needed by the chain rule to compute gradients for anything downstream. In a LoRA setup, the adapter A and B sit alongside or after a frozen base projection. During backprop the loss gradient enters the adapter's output and flows backward through both the adapter math and the base layer's contribution. To complete the chain, autograd needs the input activation that fed the base module. Without it, the local Jacobian cannot be evaluated and the gradient cannot continue backward into A and B at all. So activation memory in a LoRA run scales the same way it does in a full fine-tune. It scales with batch size, sequence length, layer count, and hidden dimension, and it is independent of whether the base weights are trainable. **Gradient checkpointing** on the base modules remains one of the highest-leverage memory levers for LoRA, especially for long-context fine-tunes where activations dominate the memory budget. The distractors are each wrong in instructive ways. B is the seductive misreading. C invents an architectural distinction that does not exist; the chain rule does not care about pre-norm versus post-norm placement of the adapter. D ties activation memory to the optimizer, but the optimizer reads gradients after they are computed and has no influence on which intermediate tensors the autograd graph must keep. The 'LoRA is small' intuition refers to weight memory and optimizer state, both of which scale with the trainable parameter count. Activation memory scales with the forward pass and remains the dominant line in long-context regimes regardless of which weights are frozen.

TL;DR

No: freezing the base weight removes its gradient but not its activations; backprop into the LoRA adapters still flows through base-layer Jacobians, so the activations must stay cached and checkpointing still pays.

Memory aid

Easy to grasp

Imagine a long relay race where only the last runner can keep the medal but every earlier runner must still hand off the baton in the right order. Saying the early runners are not eligible for the medal does not let them skip their leg of the race. The baton still has to travel through their hands or the race breaks. During the learning step, the correction signal that flows back through the network is the baton and the activations are the handoff points. Freezing a weight just means that runner does not get a medal. They still have to be in the race, still have to hand off cleanly, and the only way to do that is to remember where they were when the baton came through the first time. That is exactly why activation memory does not shrink when you freeze weights.

Key concepts

LoRA Gradient checkpointing Activation memory Backpropagation PEFT

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

LoRA's reputation as a memory-efficient fine-tuning method is well earned, but it can mislead you about which memory you actually saved. The marketing line is that LoRA lets you fine-tune billion-parameter models on consumer GPUs, and that is true. The mechanism, however, is a saving on one specific memory line, not on all of them. Understanding exactly which line LoRA collapses and which lines it leaves untouched is the difference between a run that fits comfortably and one that mysteriously OOMs at the third attention layer.

This question targets the most commonly conflated piece of that picture. Freezing a weight removes its gradient. It does not remove the activations needed to flow gradient through the layer to whatever does need them. Backprop is a chain-rule traversal of the autograd graph, and the chain rule needs the activations at every node along the path, regardless of which weights at those nodes carry gradients.

The correct answer is A. To answer with conviction you have to be able to articulate, in autograd terms, why a frozen base layer still produces an activation that backprop will consume, why the seductive option B is a precise misreading, and why the distractors C and D each encode an unrelated technical claim that does not bear on the chain rule.

Two memory lines that look the same but are not

Training a neural network spends memory on at least three distinct things, and freezing a weight only affects one of them. The first is weight memory, the storage of the parameters themselves. The second is gradient memory, the storage of per-parameter gradient buffers during the backward pass. The third is activation memory, the storage of intermediate forward-pass tensors that backward needs to compute Jacobians. The optimizer state, like Adam's first and second moment buffers, is a fourth line scaled with the trainable parameter count.

Freezing a weight changes lines two and four: no gradient buffer is allocated for that weight, and no optimizer state is maintained for it. The weight itself still exists, so line one is unchanged, and the layer's activations during the forward pass still feed downstream computation, so line three is also unchanged.

The 'LoRA is small' intuition refers to lines two and four. Trainable parameter count drops by roughly seventy times in a typical LoRA setup, so gradient and optimizer memory drop by the same factor. That is the memory win that lets a 7B fine-tune fit on a 24 GB consumer GPU. Lines one and three do not change. The base weights still occupy their full footprint, and the activations still flow through every layer as if every weight were being trained.

This is why a LoRA run on long contexts can still OOM. The forward pass produces activation tensors whose total size scales with hidden dimension, sequence length, batch size, and layer count. None of those scale with the trainable parameter count, so LoRA does nothing to reduce them. You have to attack activation memory with a separate set of tools, and the most effective of those is gradient checkpointing on the base modules.

Why the chain rule needs base-layer activations

Why each distractor is wrong

Gradient checkpointing as the right lever

Operational guidance for LoRA runs that fit

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Hugging Face PEFT documentation explicitly recommends enabling gradient_checkpointing alongside LoRA for long-context fine-tunes, addressing exactly this memory line.
Unsloth's optimized fine-tuning recipes default to gradient checkpointing on base modules even with LoRA enabled, citing the activation memory wall on 24 GB consumer GPUs.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you estimate whether activation memory or optimizer state is the binding constraint on your LoRA run?

Compute optimizer state as 8 bytes times the LoRA trainable count, weights as 2 bytes times the total parameter count (or 0.5 with NF4), and activations as roughly hidden_dim times seq_len times batch_size times layer_count times 2 bytes. The largest number is the binding constraint.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Conflating 'no gradient on the base weights' with 'no activations needed from base layers'. Those are independent properties, and only the first changes when you freeze a parameter.

60 second bullets to scan on the way to the call.

Why freezing a weight removes its gradient but not its activations
How the chain rule depends on base-layer activations even when the layer is frozen

Primary sources. Browse if you want the original framing.