Fill in the DeepSpeed config keys that select a ZeRO stage

Attempt it

In a DeepSpeed JSON config the ZeRO stage is selected by the key . , where the value is one of 1, 2, or 3. Higher stages shard more training state across GPUs at the cost of more -GPU communication.

Quick reference

TL;DR

The key is `zero_optimization.stage`, value 1, 2, or 3. Higher stages shard more training state across GPUs at the cost of more cross-GPU communication.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine eight friends carrying a giant tent to a campsite. The smallest version of the trick is each friend just carries an equal share of the tent poles, and when it is time to pitch, they pass poles around as needed. A bigger version of the trick has them split not just poles but also the assembly instructions, so each person knows their slice. The biggest version splits the tent fabric itself, so no one person ever carries the whole thing. Each step saves more weight on each friend's back, but it also means more passing things back and forth at setup time. The DeepSpeed config has one knob that names which of these three sharing levels you want.

Key concepts

DeepSpeed ZeRO sharding Data parallel

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: name the exact key path, walk through what each stage shards, then cover the communication tradeoff and when offload helps or hurts.

Real-world examples

Real products, models, and research that use this idea.

Hugging Face Accelerate and TRL ship DeepSpeed config presets at stages 1, 2, and 3 for users fine-tuning Llama 4 Maverick on multi-GPU nodes.
Microsoft's DeepSpeed examples repository uses stage 3 with parameter offload to train models that do not fit on any single GPU.
BLOOM training at the BigScience workshop used DeepSpeed ZeRO stage 1 plus pipeline parallel across hundreds of A100s.
Qwen 3.5 community fine-tunes on 4xH100 nodes typically start at stage 2 because it fits and avoids the stage 3 communication tax.
DeepSeek V4 distillation recipes use ZeRO stage 3 to materialise the teacher model across the cluster while training a smaller student adapter.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the stage 1 to stage 2 transition almost free in communication terms?

Sign in or go Pro to see the approach for this follow-up.

QWhen would CPU offload at stage 3 be slower than upgrading to a larger GPU?

Sign in or go Pro to see the approach for this follow-up.

QHow does FSDP differ from DeepSpeed ZeRO stage 3, and when would you pick one over the other?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Setting stage 3 by default without measuring throughput. The all-gather of parameters every step can dominate runtime on slow interconnects and erase the memory savings.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.