Inference is the forward pass that turns a prompt into generated tokens. Weights are frozen, no gradients are computed, no parameters are updated. Training computes gradients on labeled data and updates the weights. Inference dominates production cost because every user request hits it; training is a one-time (or occasional) amortized expense.
**LLM inference** is what happens every time you send a prompt to a model and get a reply back. The model does a forward pass: it reads the input tokens, multiplies them through its layers using the weights it already has, and produces output tokens one at a time. Nothing about the model changes during this process.
Training is what created those weights in the first place. During training, the model processes labeled examples, compares its predictions to the correct answers, computes how wrong it was (the loss), then backpropagates gradients to nudge every weight slightly toward the correct answer. Training needs gradients, an optimizer, and often three or four times more GPU memory than inference because it has to hold activations from the forward pass to use during the backward pass.
The cost picture is asymmetric. Training a frontier model can cost tens of millions of dollars but happens once. Inference is cheap per request but happens billions of times per day across all users. That is why inference optimization (faster decoding, smaller KV cache, better batching) is where most production engineering effort goes.
**Inference** is the forward-only execution of a trained model: weights are frozen, no gradients are computed, no parameters get updated. The input prompt goes through every layer, attention is computed against the cached keys and values of prior tokens, and output tokens are sampled one by one until a stop condition fires.
Training is structurally different. It requires labeled data, computes a loss against the targets, runs the backward pass to compute gradients, holds the optimizer state (typically Adam's first and second moments), and updates the weights. The memory footprint is roughly 4x the parameter count once you add activations, gradients, and optimizer state, versus roughly 2x for inference (weights plus KV cache).
The economic split is what makes this an interview question. Training a frontier model is a one-time bill measured in millions of GPU-hours. Inference is the recurring bill, growing linearly with traffic, and it dominates the lifetime cost of any deployed model. A 2-billion-parameter open-weight model might cost $100k to train but burn $5M per year on inference if it serves a large product. That asymmetry is why inference-side techniques like quantization, paged attention, continuous batching, and speculative decoding get so much engineering attention.
Quick reference
TL;DR
Inference is the forward-only pass that turns a prompt into tokens with frozen weights; training is the backward pass that updates weights on labeled data.
Picture a chef who studied for years at culinary school. The studying part is training. They read recipes, made mistakes, got corrected, and slowly learned what works. Once they leave the school, the cookbook in their head is locked. Now when you order a meal, they cook it without going back to study. That is inference: using the recipes they already learned to make one specific dish. Training happens once and is enormously expensive because they had to learn from millions of examples. Inference happens every time a customer walks in and is cheap per meal, but the restaurant serves thousands of meals a day, so the total tab adds up fast.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You are fine-tuning a model on user-specific data inside a serving loop Fine-tuning is training, not inference, and needs gradient computation, optimizer state, and a different memory budget.
You need to compute gradients for adversarial robustness testing or input attribution Standard inference frameworks disable autograd; you must explicitly re-enable it or use a training framework.
You are doing reinforcement learning with reward signals updating the policy RLHF and similar pipelines mix inference (sampling) with training (policy updates); treating it as pure inference loses the update step.
Quick scan · 2–4 min
Everything important, quickly.
4 min: forward-only vs forward plus backward, memory footprint comparison, the prefill/decode split inside inference, and why inference cost dominates the lifetime bill of any deployed model.
Side-by-side
Aspect
Training
Inference
Weight updates
Yes, every step
No, frozen
Gradients
Computed in backward pass
Never computed
Memory footprint
~4x parameter count
~2x parameter count + KV cache
Frequency
Once per model release
Every user request
Dominant bottleneck
All-reduce bandwidth across cluster
HBM bandwidth on single GPU (decode)
Real-world examples
Real products, models, and research that use this idea.
1GPT-5.5 and Claude Opus 4.7 are trained once on massive clusters and then serve billions of inference requests per day from fleets of H100 and B200 GPUs.
2vLLM, SGLang, and TensorRT-LLM are inference-only serving stacks; they hold weights, run forward passes, and never touch a backward pass.
3PyTorch's `torch.no_grad()` context manager and `model.eval()` flag are the standard way to disable gradient tracking during inference in custom serving code.
4Llama 4 Maverick's open weights can be loaded into either a training framework (FSDP, DeepSpeed) for fine-tuning or an inference framework (vLLM, llama.cpp) for serving.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does inference memory grow with the KV cache while training memory grows with activations?
Sign in or go Pro to see the approach for this follow-up.
QIf inference dominates lifetime cost, why does training still get headline attention?
Sign in or go Pro to see the approach for this follow-up.
QHow does parameter-efficient fine-tuning (LoRA) blur the inference/training line?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saying inference is 'just a smaller training run'. It is structurally different: no gradients, no optimizer state, no labels, and weights are read-only the entire time.