Part ofGenAI Engineer·Week 3: Fine-tuning & AlignmentView roadmap →

What does gradient checkpointing trade, and roughly by how much?

Flashcard·Easy·4.0 · 0·~30s·Asked atAi21 Datadog Stripe·Relevant atDatabricks

Attempt it

It trades extra compute for activation memory. The forward pass discards most intermediate activations between checkpointed segments. The backward pass re-runs the forward inside each segment to regenerate them. Typical cost is around 30 percent more wall-clock per step in exchange for roughly 60 to 70 percent less activation memory. That is usually the difference between OOM and a clean run on long sequences.

Your answer

**Gradient checkpointing** is a training technique that reduces GPU memory by trading extra compute for less activation storage. During a normal forward pass, the model keeps all intermediate activations (layer outputs, attention scores, etc.) in GPU memory. Backpropagation needs them to compute gradients. For a deep transformer with long sequences, these activations can be the largest single memory term, often larger than the weights themselves. Gradient checkpointing works by splitting the model into segments and storing activations only at the segment boundaries. Activations inside each segment are discarded as soon as the forward pass exits the segment. When the backward pass reaches a segment, the framework re-runs that segment's forward pass to regenerate the discarded activations. It then uses them to compute gradients and discards them again. The practical trade-off: - Memory savings: roughly 60 to 70 percent less activation memory for typical transformers. This is the difference between OOM (out of memory) and a clean run on long-context fine-tuning. - Compute cost: roughly 30 percent more wall-clock per step, because each segment's forward pass runs twice. It is one flag in Hugging Face Trainer (`gradient_checkpointing=True`), and TRL's SFTConfig turns it on by default.

**Gradient checkpointing trades extra compute for activation memory.** During the forward pass it discards most intermediate activations between checkpointed segment boundaries. During the backward pass it re-runs each segment's forward to regenerate the discarded activations, then uses them for gradient computation. Activation memory scales as O(L times S times H) for a transformer of L layers, S sequence length, and H hidden size. For long-sequence fine-tuning it often exceeds weight memory. A 7B model in bf16 has 14 GB of weights but well over 100 GB of activations at 8K sequence length and batch 4. Checkpointing keeps only each segment's inputs, at the cost of roughly one extra forward pass per backward. Typical magnitudes for a standard transformer with checkpointing per layer: - **Activation memory**: roughly 60 to 70 percent reduction. - **Wall-clock per step**: roughly 30 percent increase from the extra forward pass. - **Throughput in tokens per second**: roughly 20 to 25 percent decrease. The exact numbers depend on the model. Per layer segments are the default; checkpointing only some layers saves less memory and less compute. The feature composes with most other training techniques. It stacks cleanly with mixed precision, gradient accumulation, ZeRO sharding, and FSDP. It interacts subtly with FlashAttention (which has its own activation memory optimisations) and with stage-3 ZeRO (which re-gathers parameters per checkpoint, potentially redundantly). TRL's SFTConfig turns it on by default.

TL;DR

Gradient checkpointing trades extra compute for activation memory. Typical cost is roughly 30 percent more wall-clock per step in exchange for roughly 60 to 70 percent less activation memory.

Memory aid

Easy to grasp

Imagine taking a long road trip and writing down detailed notes about every turn so you can retrace your route home. Eventually your notebook fills up. Gradient checkpointing is like deciding to write down only the major waypoints (every fifth turn) instead of every single turn. When you need to find your way back, you can drive the small loops between waypoints again to reconstruct the detailed turns you did not write down. You burn extra fuel doing the small re-drives, but your notebook stays small enough to carry. For long sequences during model training, the savings are large. They are the difference between running out of GPU memory and finishing the job.

Key concepts

Gradient checkpointing Activation memory Backpropagation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Gradient checkpointing is a memory versus compute trade-off in deep learning training. The technique is conceptually simple and the implementation is a few lines. The impact on training feasibility is enormous: it routinely turns OOM crashes into successful runs at the cost of about a quarter of throughput.

The mechanism rests on a basic observation about backpropagation. The backward pass needs the activations from the forward pass to compute gradients. That is why frameworks normally keep all intermediates in memory throughout the forward. Gradient checkpointing observes that you do not have to keep them: you can drop them and recompute them on demand when backward arrives.

Activation memory: the term that gradient checkpointing targets

Training memory decomposes into four roughly orthogonal terms: weights, gradients, optimizer state, and activations. Each scales differently with model size and training configuration.

Weights, gradients, and optimizer state scale with parameter count and are fixed once the model is loaded. For a 7B model in bf16 with Adam: 14 GB weights, 14 GB gradients, 56 GB Adam moments, totaling 84 GB of fixed training state.

Activations are different. They scale with batch size, sequence length, and depth: roughly O(B times S times H times L) bytes for a transformer. For the same 7B model at batch 4 and sequence 8K, activations alone can exceed 100 GB without checkpointing, far larger than the weights.

For short sequences and small batches, weight memory dominates and activations are a minor term. For long sequences (8K and up) and reasonable batch sizes (4 and up), activations become the largest single term. Even a 7B LoRA run has fixed state of little more than the 14 GB of bf16 weights. Activations are what makes it OOM on an 80 GB H100.

Gradient checkpointing is the technique specifically designed to collapse this activation term. It does not touch weights, gradients, or optimizer state. Its entire job is to reduce the activation footprint. That is why it is the right tool for long-context fine-tuning and the wrong tool for parameter-heavy short-context training.

The recomputation mechanism in detail

Typical magnitudes and segment granularity

Composition with FlashAttention, ZeRO-3, and the reentrant flag

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Hugging Face Trainer exposes gradient_checkpointing=True as a single config flag that enables per-layer checkpointing on standard transformer architectures.
Long-context fine-tunes at 8K tokens routinely enable checkpointing, because activations at that length far outgrow the weights of a 7B base.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the compute cost of gradient checkpointing approximately 1.33x rather than 2x?

Count the passes. Without checkpointing: 1 forward + 1 backward. With checkpointing: 1 forward + 1 backward + 1 extra forward inside backward. A backward costs about twice a forward, so the baseline is about 3 forward-equivalents; one extra forward makes it 4, a ratio of 4/3.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing gradient checkpointing saves optimizer state or weight memory. It only reduces activation memory, the third memory term, by recomputing forward activations during backward.

60 second bullets to scan on the way to the call.

What memory term gradient checkpointing reduces (activations) and what it does not (weights, gradients, optimizer state)
How the recomputation mechanism works during forward and backward passes.

Primary sources. Browse if you want the original framing.