Paged optimizer in QLoRA, what gets paged, and to where?

Flashcard·Easy·4.0 · 0·~30s·Asked atFigure Ai Kore Ai Pwc·Relevant atMeta NVIDIA

Attempt it

It pages Adam's optimizer state, the first-moment buffer m and the second-moment buffer v, two values per trainable parameter, between GPU VRAM and CPU RAM under the hood via the CUDA unified-memory driver. The optimizer state stays on the CPU when the trainer is not actively running an optimizer step, then is paged back to the GPU at update time.

The practical win is OOM safety during transient memory spikes, a long sequence in the current batch, a fragmented allocator, or activation memory that grows unexpectedly will not crash the run, because the driver can evict optimizer-state pages to CPU on demand. The gradient and weight tensors are not paged; only the optimizer state. This is what lets QLoRA fit a 70B model on a single 48 GB GPU where a non-paged Adam would OOM at random points in training.

Your answer

Fine-tuning consumes GPU memory in three big chunks: the **weights**, the **activations** computed during forward and backward, and the **optimizer state** Adam carries per trainable parameter. Adam keeps two extra buffers per trainable parameter, the first moment `m` and the second moment `v`. For a 70B model with LoRA, the trainable count is small but the optimizer state still totals several gigabytes. The **paged optimizer** in QLoRA (`paged_adamw_8bit` / `paged_adamw_32bit` in bitsandbytes) keeps these buffers in **CPU RAM** by default and uses **CUDA unified memory** to migrate them to **GPU VRAM** only when the optimizer step runs. Between steps, activations and weights have the VRAM to themselves. The benefit is OOM safety. A long-sequence batch, allocator fragmentation, or an activation spike can be absorbed because the driver evicts optimizer-state pages to CPU on demand. The job survives the spike instead of crashing. Only the optimizer state is paged. Gradients and weights stay in VRAM.

**Paged optimizers** in QLoRA page Adam's per-parameter state buffers (`m` and `v`) between **GPU VRAM and CPU RAM** using the **CUDA unified memory** driver. Adam tracks two values per trainable parameter: - `m_t`: exponential moving average of gradients (first moment) - `v_t`: exponential moving average of squared gradients (second moment) For any non-trivial model these buffers add up. Even with LoRA, training a few hundred million adapter parameters in 32-bit Adam state is roughly 2 GB just for the moments. With 8-bit Adam the state shrinks but is still significant relative to the activation budget. The paged variants (`paged_adamw_8bit`, `paged_adamw_32bit`) allocate these buffers as CUDA unified-memory tensors. By default they live in CPU RAM. When the optimizer step runs, the driver migrates the pages to GPU on demand, the step updates the parameters, and the pages can be evicted again. Between optimizer steps the GPU does not hold the state. The practical gain is OOM safety. A long-sequence batch, allocator fragmentation, or an activation memory spike that would otherwise crash the run can be absorbed because the driver evicts optimizer pages on demand to free VRAM. The training job survives the transient spike at the cost of a brief pause while pages migrate. Gradients and weights are not paged. The mechanism applies only to the optimizer state, because the state is large, only needed during the optimizer step, and tolerant of the migration latency. This is the trick that lets QLoRA fit a 70B model on a single 48 GB GPU where unpaged Adam would crash at random points in training.

TL;DR

The paged optimizer keeps Adam's first- and second-moment buffers in CPU RAM and migrates them to GPU only during the optimizer step, using CUDA unified memory to avoid OOM crashes from transient VRAM spikes.

Memory aid

Easy to grasp

Picture your GPU as a small workshop with limited bench space. Three things compete for that bench: the materials you are working on, the tools you are using, and the spare parts shelf you only need once an hour. The paged optimizer is like keeping the spare parts shelf in the storage room next door (CPU RAM). When you finally need a spare part, you wheel the shelf into the workshop briefly, then push it back out so the bench is free for the actual work. The optimizer state is the spare parts: large, only needed at update time, fine to live elsewhere most of the run. The CUDA driver handles the wheeling in and out automatically. The benefit is that a brief surge in activation memory does not crash the whole job; there is breathing room because the shelf is not blocking the bench.

Key concepts

Paged optimizer QLoRA Adam optimizer Memory management

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

QLoRA fits a 70B model on a single consumer-grade GPU by combining several memory tricks. The base is 4-bit quantised, double quantised to save another small fraction, LoRA wraps only specific linears for adapter training, and the paged optimizer keeps Adam's state buffers out of GPU VRAM most of the time. Removing any one of these pieces breaks the math and the model no longer fits.

This question is about the paged optimizer specifically. The answer has two parts: what data is paged, and where it moves to. Both parts are precise. The paged data is Adam's first- and second-moment buffers per trainable parameter; the destination is CPU RAM, with on-demand migration to GPU VRAM during the optimizer step, mediated by the CUDA unified memory driver.

This deep dive walks through what optimizer state actually consists of, why it is the right tenant to page rather than weights or activations, how unified memory mechanically performs the migration, the OOM safety benefit that makes the technique valuable, the throughput cost on consumer hardware, and what is not paged. By the end the role of the paged optimizer in the QLoRA recipe should look like a precise architectural choice rather than a vague memory trick.

What lives in Adam's state and why it is large

Adam-family optimizers track two values per trainable parameter beyond the parameter itself.

The first moment

m_t is an exponential moving average of past gradients. It captures the recent direction of gradient flow and is what Adam uses to compute the smoothed update direction.

The second moment

v_t is an exponential moving average of past squared gradients. It estimates the per-parameter gradient variance and is used to normalise the update so noisier parameters take smaller steps.

The size of these buffers

For a trainable parameter count of N, Adam's state is 2N values. In 32-bit precision (the default AdamW), that is 8N bytes. In 8-bit precision (adamw_8bit from bitsandbytes), it shrinks to 2N bytes. For a typical QLoRA fine-tune of Llama 3.1 8B with rank 16 adapters on all attention and MLP linears, N is roughly 50M trainable parameters. Adam state in 32-bit is 400MB; in 8-bit it is 100MB. For a 70B model with similar adapter coverage, those numbers scale to several GB.

Why this matters

A single 80GB GPU running QLoRA on Llama 3.1 70B has roughly: 35GB for the 4-bit quantised base weights, 5 to 15GB for activations depending on sequence length and batch size, and several GB for Adam state. Add small overhead for the LoRA adapter weights, gradients, and the CUDA allocator's bookkeeping, and you are already near the budget.

For smaller GPUs (24GB to 48GB) the math is tighter still. Adam state competing with activations is a real source of OOM crashes, particularly when a long-sequence batch pushes activation memory above its expected peak.

Why optimizer state is the right tenant to page

How CUDA unified memory mechanically performs the migration

OOM safety, throughput cost, and what is not paged

Putting it together

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Hugging Face Transformers Trainer exposes paged_adamw_8bit and paged_adamw_32bit via the optim argument, the standard choice for QLoRA recipes on Llama 3.1 and Mistral.
Unsloth tutorials demonstrate paged_adamw_8bit for fine-tuning Llama 3.1 8B on a single 24GB consumer GPU, leaning on the paging to survive long-sequence batches.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the throughput cost of using a paged optimizer compared to an unpaged one, and where does that cost come from?

Quantify the cost as the time to migrate optimizer state pages over PCIe per optimizer step; estimate based on bandwidth and state size, and explain why steady state cost is small but worst-case spikes are larger.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing the paged optimizer reduces total memory consumption. It actually only shifts where Adam's state lives; the win is OOM safety during transient spikes, not absolute footprint reduction.

60 second bullets to scan on the way to the call.

What data the paged optimizer moves between CPU and GPU
Which CUDA mechanism backs the migration

Primary sources. Browse if you want the original framing.