Inference is the forward-only pass that turns a prompt into tokens with frozen weights; training is the backward pass that updates weights on labeled data.
Picture a chef who studied for years at culinary school. The studying part is training. They read recipes, made mistakes, got corrected, and slowly learned what works. Once they leave the school, the cookbook in their head is locked. Now when you order a meal, they cook it without going back to study. That is inference: using the recipes they already learned to make one specific dish. Training happens once and is enormously expensive because they had to learn from millions of examples. Inference happens every time a customer walks in and is cheap per meal, but the restaurant serves thousands of meals a day, so the total tab adds up fast.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every LLM has two completely different lives. There is the life it lives once, in a giant cluster, learning what it knows. Then there is the life it lives a billion times a day, on whatever GPUs are cheapest to rent, turning prompts into tokens. The first life is training. The second is inference. Despite sharing the same neural network architecture and most of the same forward-pass code, these two phases are different problems with different bottlenecks, different memory budgets, different infrastructure, and different teams optimizing them.
This deep dive walks through what actually happens during each phase, why their memory footprints differ by roughly 2x even before you add optimizer state, why inference splits into a parallel prefill and a sequential decode, and why the economic shape of inference dominates the cost of any deployed model. By the end, you should be able to answer not just 'what is inference' but also 'why does inference need its own optimization stack at all'.
The compute graph: forward only vs forward plus backward
Training and inference both run a forward pass through the network. Embeddings, attention, feed-forward layers, layer norms, the output projection. That graph is identical in structure. What changes is everything bolted onto it.
During training, the forward pass is followed by a backward pass that propagates the loss gradient backward through every operation using the chain rule. Each layer computes how much each of its weights contributed to the loss, then the optimizer takes a small step in the direction that reduces the loss. The backward pass is roughly 2x the FLOPs of the forward pass, because each operation has to compute gradients with respect to both its inputs and its parameters.
During inference, none of that happens. The forward pass runs in torch.no_grad() mode (or its framework equivalent), which means PyTorch does not build the autograd graph. No activations need to be kept around after they are consumed by the next layer. The model is read-only the entire time. The output of the forward pass is logits, which a sampling step turns into the next token, and that loop repeats until a stop condition fires.
This difference cascades everywhere. Inference frameworks like vLLM and TensorRT-LLM are written from the start without autograd machinery. They use kernel fusion, custom CUDA, and integer-arithmetic tricks that would break gradient flow if you tried to run them in training mode. The two stacks share the same model architecture and the same weight files, but the engineering inside them barely overlaps.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Training | Inference |
|---|---|---|
| Weight updates | Yes, every step | No, frozen |
| Gradients | Computed in backward pass | Never computed |
| Memory footprint | ~4x parameter count | ~2x parameter count + KV cache |
| Frequency | Once per model release | Every user request |
| Dominant bottleneck | All-reduce bandwidth across cluster | HBM bandwidth on single GPU (decode) |
Real products, models, and research that use this idea.
- GPT-5.5 and Claude Opus 4.7 are trained once on massive clusters and then serve billions of inference requests per day from fleets of H100 and B200 GPUs.
- vLLM, SGLang, and TensorRT-LLM are inference-only serving stacks; they hold weights, run forward passes, and never touch a backward pass.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does inference memory grow with the KV cache while training memory grows with activations?
Inference only needs to keep K and V projections of past tokens so future attention can read them without recomputing. Training must also hold every layer's forward activations to multiply with incoming gradients during backprop. Both are quadratic-ish in sequence length but for different reasons.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying inference is 'just a smaller training run'. It is structurally different: no gradients, no optimizer state, no labels, and weights are read-only the entire time.
60 second bullets to scan on the way to the call.
Definition of inference as a forward-only pass with frozen weights
What 'no gradients' means at the operation level
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.