Gradient checkpointing, what does it trade, and roughly by how much?

Attempt it

It trades extra compute for activation memory. The forward pass discards most intermediate activations between checkpointed segments; the backward pass re-runs the forward inside each segment to regenerate them. Typical cost is around 30 percent more wall-clock per step in exchange for roughly 60 to 70 percent less activation memory, usually the difference between OOM and a clean run on long sequences.

Your answer

**Gradient checkpointing** is a training-time technique that reduces GPU memory by trading extra compute for less activation storage. The baseline situation: during a normal forward pass, the model keeps all intermediate activations (layer outputs, attention scores, etc.) in GPU memory because backpropagation needs them to compute gradients. For a deep transformer with long sequences, these activations can be the largest single memory term, often larger than the weights themselves. Gradient checkpointing works by partitioning the model into segments and storing activations only at the segment boundaries. Activations inside each segment are discarded as soon as the forward pass exits the segment. When the backward pass reaches a segment, the framework re-runs that segment's forward pass to regenerate the discarded activations on demand, then uses them to compute gradients, then discards them again. The practical tradeoff: - Memory savings: roughly 60 to 70 percent less activation memory for typical transformer setups. This is the difference between OOM and a clean run on long context fine-tuning. - Compute cost: roughly 30 percent more wall-clock per step, because each segment's forward pass runs twice (once during the original forward, once during the recomputed backward). The feature is essentially free to enable in modern frameworks (`gradient_checkpointing=True` in Hugging Face Trainer) and is almost always on for any fine-tuning run with sequence length above 2048 or model size above 7B.

**Gradient checkpointing trades extra compute for activation memory.** During the forward pass it discards most intermediate activations between checkpointed segment boundaries; during the backward pass it re-runs each segment's forward to regenerate the discarded activations, then uses them for gradient computation. The motivation is that activation memory scales as O(L times S times H) for a transformer of L layers, S sequence length, and H hidden size, often exceeding weight memory for long-sequence fine-tuning. A 7B model in bf16 has 14 GB of weights but easily 30+ GB of activations at 8K sequence length and batch 4. Checkpointing collapses the activation term by a factor of roughly the segment count, at the cost of roughly one extra forward pass per backward. Typical magnitudes for a standard transformer with per-layer checkpointing: - **Activation memory**: roughly 60 to 70 percent reduction in peak training memory. - **Wall-clock per step**: roughly 30 percent increase from the extra forward pass. - **Throughput in tokens per second**: roughly 20 to 25 percent decrease. The exact numbers depend on the model and the checkpoint granularity. Coarser segments (one checkpoint per N layers) save less memory but cost less compute; finer segments (one per layer, the default in most frameworks) maximise memory savings at the larger compute cost. The feature composes with most other training techniques. It stacks cleanly with mixed precision, gradient accumulation, ZeRO sharding, and FSDP. It interacts subtly with FlashAttention (which has its own activation-memory optimisations) and with stage-3 ZeRO (which re-gathers parameters per checkpoint, potentially redundantly). In modern fine-tuning recipes for any model above 7B at any sequence length above 2K, gradient checkpointing is on by default.

Quick reference

TL;DR

Gradient checkpointing trades extra compute for activation memory. Typical cost is roughly 30 percent more wall-clock per step in exchange for roughly 60 to 70 percent less activation memory.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine taking a long road trip and writing down detailed notes about every turn so you can retrace your route home. Eventually your notebook fills up. Activation checkpointing is like deciding to write down only the major waypoints (every fifth turn) instead of every single turn. When you need to find your way back, you can drive the small loops between waypoints again to reconstruct the detailed turns you did not write down. You burn extra fuel doing the small re-drives, but your notebook stays small enough to carry. For long sequences during model training, the savings are dramatic: the difference between running out of GPU memory and finishing the job.

Key concepts

Gradient checkpointing Activation memory Backpropagation

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: explain the compute-memory tradeoff, walk through the recomputation mechanism, then cover the typical magnitudes and the interactions with FlashAttention and ZeRO-3.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face Trainer exposes gradient_checkpointing=True as a single config flag that enables per-layer checkpointing on standard transformer architectures.
Production Llama 4 Maverick fine-tunes at 8K context routinely enable checkpointing to fit activations alongside the 70B base on 4xH100.
Axolotl and Unsloth both ship YAML configs with gradient_checkpointing on by default for any model above 7B.
PyTorch FSDP uses torch.utils.checkpoint.checkpoint with use_reentrant=False under the hood for activation checkpointing in distributed training.
Qwen 3.5 32B QLoRA recipes on consumer 24GB GPUs depend on gradient checkpointing to make the activation footprint fit alongside the 4-bit base and LoRA adapters.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the compute cost of gradient checkpointing approximately 1.33x rather than 2x?

Sign in or go Pro to see the approach for this follow-up.

QHow does gradient checkpointing compose with FlashAttention, and why does the memory saving shrink?

Sign in or go Pro to see the approach for this follow-up.

QWhen would coarser checkpoint granularity (every 4 layers instead of every 1 layer) be the right choice?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing gradient checkpointing saves optimizer state or weight memory. It only reduces activation memory, the third memory term, by recomputing forward activations during backward.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.