The correct answer is **A**. Freezing the base weights does not remove activations from the picture, and **gradient checkpointing** on those base modules still pays off.
Here is the core distinction. Freezing a weight means the optimizer never updates it, so no gradient needs to be computed for that weight specifically. That saves the gradient buffer on those weights. But the chain rule still requires the activations of each layer to compute gradients for anything downstream that does need them, and that includes the LoRA adapters sitting on top.
When the loss gradient flows backward, it passes through every base-layer operation that produced the activation feeding the LoRA adapter. Each of those operations has a local Jacobian that depends on its input. Without the cached input activation, the Jacobian cannot be evaluated, so the gradient cannot continue backward into the adapter.
The other options are wrong for specific reasons. B confuses no gradient on weight with no activations needed. C invents a pre-norm versus post-norm distinction that the chain rule does not respect. D invents an optimizer-specific rule, but activation memory is decided by the autograd graph, not by which optimizer reads the gradients afterward.
The practical takeaway: LoRA runs on long contexts still hit activation memory walls, and checkpointing on the base modules is one of the most useful levers for fitting them.
**The answer is A**, and the right framing is to separate two distinct concepts that the trap option B fuses incorrectly. Freezing a weight removes gradient computation for that weight. It does not remove the requirement to cache the activations needed by the chain rule to compute gradients for anything downstream.
In a LoRA setup, the adapter A and B sit alongside or after a frozen base projection. During backprop the loss gradient enters the adapter's output and flows backward through both the adapter math and the base layer's contribution. To complete the chain, autograd needs the input activation that fed the base module. Without it, the local Jacobian cannot be evaluated and the gradient cannot continue backward into A and B at all.
So activation memory in a LoRA run scales the same way it does in a full fine-tune. It scales with batch size, sequence length, layer count, and hidden dimension, and it is independent of whether the base weights are trainable. **Gradient checkpointing** on the base modules remains one of the highest-leverage memory levers for LoRA, especially for long-context fine-tunes where activations dominate the memory budget.
The distractors are each wrong in instructive ways. B is the seductive misreading. C invents an architectural distinction that does not exist; the chain rule does not care about pre-norm versus post-norm placement of the adapter. D ties activation memory to the optimizer, but the optimizer reads gradients after they are computed and has no influence on which intermediate tensors the autograd graph must keep.
The 'LoRA is small' intuition refers to weight memory and optimizer state, both of which scale with the trainable parameter count. Activation memory scales with the forward pass and remains the dominant line in long-context regimes regardless of which weights are frozen.
Quick reference
TL;DR
No: freezing the base weight removes its gradient but not its activations; backprop into the LoRA adapters still flows through base-layer Jacobians, so the activations must stay cached and checkpointing still pays.
Imagine a long relay race where only the last runner can keep the medal but every earlier runner must still hand off the baton in the right order. Saying the early runners are not eligible for the medal does not let them skip their leg of the race. The baton still has to travel through their hands or the race breaks. During the learning step, the correction signal that flows back through the network is the baton and the activations are the handoff points. Freezing a weight just means that runner does not get a medal. They still have to be in the race, still have to hand off cleanly, and the only way to do that is to remember where they were when the baton came through the first time. That is exactly why activation memory does not shrink when you freeze weights.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very short-sequence fine-tunes at small batch sizes where activations are not the memory bottleneck When the forward pass produces little activation memory, gradient checkpointing's compute cost outweighs its memory savings.
Inference-only deployments where no backward pass runs Activations only need to be retained for backprop; serving stacks discard them immediately and do not need checkpointing.
Tasks where the LoRA adapter is attached only to a small subset of layers at the bottom of the model Activations for layers above the highest adapter still must be cached to flow gradient down, but the chain shortens; checkpointing's value scales with the depth covered.
Quick scan · 2–4 min
Everything important, quickly.
5 min: pick A, separate weight-memory from activation-memory, walk through why base-layer activations are still needed for chain-rule backprop into the adapters, and explain why each distractor encodes a different misreading.
Real-world examples
Real products, models, and research that use this idea.
1Hugging Face PEFT documentation explicitly recommends enabling gradient_checkpointing alongside LoRA for long-context fine-tunes, addressing exactly this memory line.
2Unsloth's optimized fine-tuning recipes default to gradient checkpointing on base modules even with LoRA enabled, citing the activation memory wall on 24 GB consumer GPUs.
3FlashAttention 2's online-softmax recomputation cuts attention-block activation memory independently of LoRA, making 8k+ context LoRA runs practical on a single GPU.
4Axolotl, the LLM fine-tuning framework used for many open-weight Llama 4 and Qwen 3.5 derivatives, pairs LoRA with gradient checkpointing by default in its long-context recipes.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you estimate whether activation memory or optimizer state is the binding constraint on your LoRA run?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the wall-clock cost of gradient checkpointing on a 32-layer model, and how does it compare to the memory saving?
Sign in or go Pro to see the approach for this follow-up.
QHow does FlashAttention 2 interact with gradient checkpointing on a LoRA run?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Conflating 'no gradient on the base weights' with 'no activations needed from base layers'. Those are independent properties, and only the first changes when you freeze a parameter.