Paged optimizer in QLoRA, what gets paged, and to where?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

It pages Adam's optimizer state, the first-moment buffer m and the second-moment buffer v, two values per trainable parameter, between GPU VRAM and CPU RAM under the hood via the CUDA unified-memory driver. The optimizer state stays on the CPU when the trainer is not actively running an optimizer step, then is paged back to the GPU at update time.

The practical win is OOM safety during transient memory spikes, a long sequence in the current batch, a fragmented allocator, or activation memory that grows unexpectedly will not crash the run, because the driver can evict optimizer-state pages to CPU on demand. The gradient and weight tensors are not paged; only the optimizer state. This is what lets QLoRA fit a 70B model on a single 48 GB GPU where a non-paged Adam would OOM at random points in training.

Your answer

Fine-tuning consumes GPU memory in three big chunks: the **weights**, the **activations** computed during forward and backward, and the **optimizer state** Adam carries per trainable parameter. Adam keeps two extra buffers per trainable parameter, the first moment `m` and the second moment `v`. For a 70B model with LoRA, the trainable count is small but the optimizer state still totals several gigabytes. The **paged optimizer** in QLoRA (`paged_adamw_8bit` / `paged_adamw_32bit` in bitsandbytes) keeps these buffers in **CPU RAM** by default and uses **CUDA unified memory** to migrate them to **GPU VRAM** only when the optimizer step runs. Between steps, activations and weights have the VRAM to themselves. The benefit is OOM safety. A long-sequence batch, allocator fragmentation, or an activation spike can be absorbed because the driver evicts optimizer-state pages to CPU on demand. The job survives the spike instead of crashing. Only the optimizer state is paged. Gradients and weights stay in VRAM.

**Paged optimizers** in QLoRA page Adam's per-parameter state buffers (`m` and `v`) between **GPU VRAM and CPU RAM** using the **CUDA unified memory** driver. Adam tracks two values per trainable parameter: - `m_t`: exponential moving average of gradients (first moment) - `v_t`: exponential moving average of squared gradients (second moment) For any non-trivial model these buffers add up. Even with LoRA, training a few hundred million adapter parameters in 32-bit Adam state is roughly 2 GB just for the moments. With 8-bit Adam the state shrinks but is still significant relative to the activation budget. The paged variants (`paged_adamw_8bit`, `paged_adamw_32bit`) allocate these buffers as CUDA unified-memory tensors. By default they live in CPU RAM. When the optimizer step runs, the driver migrates the pages to GPU on demand, the step updates the parameters, and the pages can be evicted again. Between optimizer steps the GPU does not hold the state. The practical gain is OOM safety. A long-sequence batch, allocator fragmentation, or an activation memory spike that would otherwise crash the run can be absorbed because the driver evicts optimizer pages on demand to free VRAM. The training job survives the transient spike at the cost of a brief pause while pages migrate. Gradients and weights are not paged. The mechanism applies only to the optimizer state, because the state is large, only needed during the optimizer step, and tolerant of the migration latency. This is the trick that lets QLoRA fit a 70B model on a single 48 GB GPU where unpaged Adam would crash at random points in training.

Quick reference

TL;DR

The paged optimizer keeps Adam's first- and second-moment buffers in CPU RAM and migrates them to GPU only during the optimizer step, using CUDA unified memory to avoid OOM crashes from transient VRAM spikes.

Memory aid

Easy to grasp

Picture your GPU as a small workshop with limited bench space. Three things compete for that bench: the materials you are working on, the tools you are using, and the spare parts shelf you only need once an hour. The paged optimizer is like keeping the spare parts shelf in the storage room next door (CPU RAM). When you finally need a spare part, you wheel the shelf into the workshop briefly, then push it back out so the bench is free for the actual work. The optimizer state is the spare parts: large, only needed at update time, fine to live elsewhere most of the run. The CUDA driver handles the wheeling in and out automatically. The benefit is that a brief surge in activation memory does not crash the whole job; there is breathing room because the shelf is not blocking the bench.

Key concepts

Paged optimizer QLoRA Adam optimizer Memory management

Detailed answer & concept explanation

~11 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: name Adam's m and v buffers, place the paging between CPU RAM and GPU VRAM via CUDA unified memory, explain why optimizer state is the right tenant to page, and call out OOM safety as the win and PCIe bandwidth as the cost.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face Transformers Trainer exposes paged_adamw_8bit and paged_adamw_32bit via the optim argument, the standard choice for QLoRA recipes on Llama 3.1 and Mistral.
Unsloth tutorials demonstrate paged_adamw_8bit for fine-tuning Llama 3.1 8B on a single 24GB consumer GPU, leaning on the paging to survive long-sequence batches.
Axolotl QLoRA configs for DeepSeek V4 and Gemma 4 default to paged_adamw_32bit to combine bitsandbytes optimizer paging with the rest of the QLoRA recipe.
Tim Dettmers's bitsandbytes documentation walks through the unified memory mechanism that backs the paged variants and how it interacts with CUDA allocator behavior.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the throughput cost of using a paged optimizer compared to an unpaged one, and where does that cost come from?

QHow does the paged optimizer differ from CPU offloading approaches like DeepSpeed ZeRO-Offload?

QWhy is optimizer state the right tenant to page rather than activations or weights?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing the paged optimizer reduces total memory consumption. It actually only shifts where Adam's state lives; the win is OOM safety during transient spikes, not absolute footprint reduction.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.