Accelerate launch: what does that one CLI actually wire up?
Three jobs: spawn one process per GPU, initialise the distributed backend with rank and world-size env vars, and apply runtime plugins like mixed precision and FSDP or DeepSpeed sharding.
Picture a relay race. Plain Python is one runner who runs the whole track alone. The Accelerate launcher is the race organiser. Before anyone runs, the organiser walks onto the field, lines up the right number of runners, hands each one a numbered bib so they know their lane, sets up the baton handoff zones, and tells everyone which microphone the coach will be shouting through. Only then does the race start. The runners do not need to know how the bibs were printed or who set up the radios. They just see their number, their lane, and the start signal. That is what the launcher does for training processes: spawn them, number them, wire them together, and apply any race-day rules before the training loop begins.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Hugging Face's Accelerate launcher is a small command that does a lot of plumbing. To the user it looks like a thin wrapper around python train.py. Internally it is the bridge from a single-process Python invocation to a fully configured distributed training world. The reason the wrapper exists is that there is a large gap between what a Python interpreter does on its own and what a distributed training job needs before its first forward pass.
A distributed run requires three classes of setup that plain Python cannot do. The right number of processes have to be spawned on the right hosts. Those processes have to discover each other and agree on rank assignments and collective communication endpoints. And any runtime features that change how the training loop executes, like mixed precision or model sharding, have to be applied before the script wraps its model. None of these are part of the user's train.py. All of them are part of the launcher's job.
The payoff for putting this work in the launcher rather than in the script is uniformity. The same Accelerator() call inside train.py works whether the launcher started one process on one GPU, eight processes on one node with FSDP wrapping, or sixty-four processes across eight nodes under DeepSpeed ZeRO-3. The training code never has to branch on the deployment shape. This deep dive walks through each of the launcher's three responsibilities and ends with the practical consequences for debugging and scaling.
Process spawning: from one process to one per GPU
A plain python train.py invocation produces exactly one process. That process can use one GPU through model.to('cuda') or, with torch.nn.DataParallel, can shard a batch across GPUs from a single process. Modern distributed training has moved away from single-process multi-GPU for both performance and correctness reasons; the standard pattern is one process per GPU.
The launcher's first job is to spawn that one-per-GPU process group. It reads the accelerate config file, which the user sets up once with accelerate config. The config records the deployment shape: number of machines, number of processes per machine, the launcher backend, and the distributed type. The launcher then either fork-execs the right number of Python interpreters itself or shells out to an underlying tool like torchrun that does so.
On a single node the underlying launcher is typically the torchrun-equivalent that ships with Accelerate. On multi-node setups it can be MPI or a SLURM-aware launcher that handles host discovery through the scheduler's environment. The user does not need to know which one; the config drives the choice. The training script never sees this layer. It only sees the post-spawn environment.
The spawning behaviour is what makes accelerate launch non-optional for multi-GPU runs. You cannot replace it with a shell loop or a process pool because the spawned processes need shared environment variables that have to be computed before any of them start. That computation is the launcher's responsibility.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face TRL and PEFT examples ship with `accelerate launch` as the canonical entry point for multi-GPU Llama 4 and Qwen 3.5 fine-tuning recipes.
- Axolotl and LLaMA-Factory both expose `accelerate launch` wrappers so the same YAML config drives single-GPU dev and multi-node production runs.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is the difference between global rank and local rank, and why does the launcher need both?
Global rank is unique across the whole job and is used to identify a process inside the NCCL world. Local rank is unique only within a single node and selects the GPU index on that node. The launcher needs global rank for collective communication and local rank to assign each process to a specific local GPU device.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking the launcher is just a fancy wrapper that runs your script faster. It does no work inside your training loop; it sets up the world before the script starts.
60 second bullets to scan on the way to the call.
The three responsibilities: spawning, backend init, runtime plugins
Why plain Python cannot do distributed training on its own
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.