Hugging Face's accelerate launch train.py: what does the launcher orchestrate that plain python train.py does not?
Three jobs: spawn one process per GPU, set up the distributed backend through rank and world-size env vars, and apply runtime plugins like mixed precision and FSDP or DeepSpeed sharding.
Picture a relay race. Plain Python is one runner who runs the whole track alone. The Accelerate launcher is the race organiser. Before anyone runs, the organiser walks onto the field and lines up the right number of runners. The organiser hands each one a numbered bib so they know their lane. Then the organiser sets up the baton handoff zones and tells everyone which microphone the coach will be shouting through. Only then does the race start. The runners do not need to know how the bibs were printed or who set up the radios. They just see their number, their lane, and the start signal. That is what the launcher does for training processes: spawn them, number them, wire them together, and apply any race-day rules before the training loop begins.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Hugging Face's Accelerate launcher is a small command that does a lot of plumbing. To the user it looks like a thin wrapper around python train.py. Internally it is the bridge from a single-process Python invocation to a fully configured distributed training world. The wrapper exists because of a large gap. On one side is what a Python interpreter does on its own. On the other is what a distributed training job needs before its first forward pass.
A distributed run requires three classes of setup that plain Python cannot do. The right number of processes have to be spawned on the right hosts. Those processes have to discover each other and agree on rank assignments and collective communication endpoints. And any runtime features that change how the training loop executes, like mixed precision or model sharding, have to be applied before the script wraps its model. None of these are part of the user's train.py. All of them are part of the launcher's job.
The payoff for putting this work in the launcher rather than in the script is uniformity. The same Accelerator() call inside train.py works whether the launcher started one process on one GPU, eight processes on one node with FSDP wrapping, or sixty-four processes across eight nodes under DeepSpeed ZeRO-3. The training code never has to branch on the deployment shape.
Process spawning: from one process to one per GPU
A plain python train.py invocation produces exactly one process. That process can use one GPU through model.to('cuda') or, with torch.nn.DataParallel, can shard a batch across GPUs from a single process. Modern distributed training has moved away from single-process multi-GPU for both performance and correctness reasons. The standard pattern is one process per GPU.
The launcher's first job is to spawn that one-per-GPU process group. It reads the accelerate config file, which the user sets up once with accelerate config. The config records the deployment shape: number of machines, number of processes per machine, the launcher backend, and the distributed type. The launcher then either fork-execs the right number of Python interpreters itself or shells out to an underlying tool like torchrun that does so.
On a single node the underlying launcher is typically PyTorch's torchrun machinery (torch.distributed.run). On multi-node setups accelerate launch runs once per node with that node's machine rank, often under a scheduler such as SLURM. DeepSpeed runs can use DeepSpeed's own multi-node launcher. The user does not need to know which one. The config drives the choice. The training script never sees this layer. It only sees the post-spawn environment.
The spawned processes need consistent environment variables that have to be computed before any of them start. accelerate launch is the simplest way to get them right for multi-GPU runs. torchrun can also start an Accelerate script. A hand-written shell loop works only if it sets those variables correctly for every process.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Hugging Face TRL and PEFT examples ship with accelerate launch as the standard entry point for multi-GPU fine-tuning setups.
- Axolotl and LLaMA-Factory both expose accelerate launch wrappers so the same YAML config drives single-GPU dev and multi-node production runs.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is the difference between global rank and local rank, and why does the launcher need both?
Global rank is unique across the whole job and is used to identify a process inside the NCCL world. Local rank is unique only within a single node and selects the GPU index on that node. The launcher needs global rank for collective communication and local rank to assign each process to a specific local GPU device.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking the launcher is just a fancy wrapper that runs your script faster. It does no work inside your training loop. It sets up the world before the script starts.
60 second bullets to scan on the way to the call.
The three responsibilities: spawning, backend init, runtime plugins
Why plain Python cannot do distributed training on its own
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.