QLoRA OOMs at step 7000 after one clean epoch on a 24 GB GPU: which non-obvious causes deserve a look?

Multi-select·Medium·4.0 · 0·~1 min·Asked atLtimindtree NVIDIA Tech Mahindra·Relevant atDatabricks Meta Microsoft

Attempt it

The real culprits are A, B, and E. Each one matches the failure pattern of OOM at step 7000 after a clean first epoch. A is about sequence packing. The trainer concatenates short examples up to max_seq_len to use the GPU efficiently. The lengths of packed sequences vary; most packs are smaller than the cap, but the distribution has a long tail. After 7000 steps you have drawn enough packs to eventually hit one near the cap. Peak activation memory for that pack spikes well above the typical step, and a card already running near its limit tips over. B is about eval. If the periodic eval loop runs without torch.no_grad or inference_mode, every forward pass during eval builds an autograd graph as if you were about to backprop. That transient graph eats memory the same way a training forward does, doubling working memory during eval. On a tight card the eval spike is what OOMs. E is about logging. If you accumulate per-step losses in a Python list without calling .detach() or .item() first, every loss tensor keeps its autograd graph alive. Memory grows linearly with steps. After 7000 steps you have 7000 graphs in memory, and eventually that bumps the card over. C and D are wrong. Frameworks cache base weights locally and do not silently re-download mid-run. AdamW allocates its m and v buffers once at the first optimizer step and reuses them; state does not grow per step.

**Correct: A, B, E.** Each matches the failure signature (clean first epoch, OOM mid-second epoch). **A (packed-sequence spike).** Sequence packing concatenates short examples up to max_seq_len. The pack-length distribution has a long tail; enough samples eventually produce a near-cap pack. Peak attention activations scale O(L^2), so a long pack spikes far above the average. After 7000 steps you have drawn enough samples to hit one. **B (eval without inference_mode).** Forward passes inside torch.no_grad or torch.inference_mode skip autograd graph construction. Without that wrapper, every forward during eval transiently allocates activation memory as if a backward will follow, doubling working memory at the eval moment. **E (undetached loss accumulation).** Storing loss tensors in a Python list without `.detach()` or `.item()` keeps the autograd graph attached. Memory grows linearly with steps. After thousands of steps the cumulative graph mass becomes significant. **C is wrong** because Hugging Face caches weights on first load and serves from cache; there is no silent re-download mid-run. **D is wrong** because AdamW allocates m and v buffers once at the first optimizer step and reuses them. State is constant after step 1. Diagnostic: plot peak memory per step. Monotonic growth points at E; rare spikes at A; cadence-aligned spikes at B.

TL;DR

A, B, and E are the real causes: rare packed-sequence spikes (A), eval without inference_mode (B), and undetached loss accumulation (E). C does not happen; D misdescribes AdamW.

Memory aid

Easy to grasp

Picture a backpack you can just barely close. Most days it holds the same kit of items and zips fine. Then one rare day a friend hands you a long umbrella that does not normally fit, and the backpack will not close. Or you start carrying a small souvenir from every day on top of the regular kit, and after enough days the backpack overflows. Or one afternoon you decide to also pack a duplicate of your bag inside it temporarily, and that is too much weight at once. Three of the listed problems are stories like these. The other two are not really problems at all; they describe things that never actually happen in the system.

Key concepts

QLoRA Memory debugging Sequence packing Training stability

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Mid-run OOMs have a different debugging structure than first-step OOMs. A first-step OOM is constant: something is sized wrong from the start, and the fix is to resize. A mid-run OOM is either slow-growth (a leak that accumulates across steps) or a rare spike (input variability that eventually draws a worst-case sample). The arrival time at step 7000 rules out constant sizing and rules in growth or spike.

The question gives five candidate causes. Three of them are real failure modes (A: sequence-packing spike, B: eval without inference_mode, E: undetached loss accumulation). Two are not (C: silent re-download, D: AdamW growth). Knowing the difference requires a precise understanding of how each subsystem actually behaves.

This deep dive walks through the diagnostic framework, dissects each candidate cause, and closes with the instrumentation pattern that makes future mid-run OOMs much faster to diagnose.

The diagnostic framework: constant, growth, or spike

Every OOM in a training loop falls into one of three families based on its time signature.

Constant OOMs appear immediately, at step 1 or even at model initialization. The cause is sizing: the batch is too large, the model does not fit, sequence length is too long for the available memory. The fix is to resize (smaller batch, lower precision, gradient checkpointing, model sharding).

Growth OOMs appear at a step number that scales with how fast memory is leaking. The signature is monotonic growth in memory_allocated over steps. The cause is something in the training loop keeping references to tensors that should be freed: undetached loss tensors, accumulated metric tensors, a list that never gets cleared.

Spike OOMs appear at an unpredictable late step, after which the same recipe might run cleanly for hundreds of steps before spiking again. The signature is sudden peak memory at the OOM step that is much higher than recent average. The cause is input variability: a longer-than-average sequence, a larger-than-average batch, or a code path that runs rarely but allocates a lot.

The step-7000 OOM after a clean first epoch is either growth or spike. That immediately rules out the constant family. Among the five options in the question, A is a spike cause, B is a periodic spike cause (aligned with eval cadence), and E is a growth cause. C is impossible because Hugging Face does not re-download mid-run, and D is impossible because AdamW state is constant after step 1.

A: sequence-packing spike

B: eval without inference_mode

E: undetached loss accumulation

Why C and D are wrong

Instrumentation that catches all three

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Failure mode	Time signature	Diagnostic signal	Fix
A: long-pack spike	Random late step	Peak memory correlates with max seq length	Pre-allocate worst-case or cap pack length
B: eval graph	On eval step multiples	OOM aligns with eval cadence	Wrap eval in torch.inference_mode()
E: undetached leak	Monotonic growth	memory_allocated climbs steadily	Call .detach() or .item() on logged tensors
C: re-download (wrong)	n/a	Does not happen	n/a
D: optimizer growth (wrong)	n/a	Constant after step 1	n/a

Real products, models, and research that use this idea.

Hugging Face TRL SFTTrainer documentation explicitly warns about sequence packing producing long-tail packs, which is the canonical setup for the A failure mode.
PyTorch documentation for torch.inference_mode covers the eval-time memory implications and recommends it over torch.no_grad for inference and eval loops.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you instrument a training loop to distinguish A, B, and E in practice?

Log three signals every step: torch.cuda.memory_allocated, torch.cuda.max_memory_allocated since last log, and the input sequence length. Plot all three over steps. Monotonic memory_allocated growth points at E. Spikes in max_memory_allocated correlated with long sequences point at A. Spikes aligned with eval cadence point at B.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Picking D because AdamW state sounds like a growth source, or picking C because re-downloads sound plausible. Neither describes actual framework behavior; the real culprits are spikes (A), eval-time graph builds (B), and undetached accumulation (E).

60 second bullets to scan on the way to the call.

Why peak memory under sequence packing is determined by the longest pack
How the absence of inference_mode doubles eval-time working memory

Primary sources. Browse if you want the original framing.