Which scenario most justifies the extra cost of QAT over PTQ for inference quantization?
QAT only earns its training cost in the sub-4-bit regime where PTQ accuracy collapses and the deployment cannot tolerate that loss; otherwise cheap PTQ wins.
Imagine shrinking a detailed painting onto a tiny postage stamp. PTQ paints the full canvas first, then shrinks it and hopes the picture survives. For a mild shrink that works fine. But shrink it really hard and the picture turns to mush, because the painter never planned for so little space. QAT lets the painter practice on the tiny stamp the whole time, so they learn to place every stroke where it still reads clearly after shrinking. That practice costs a lot of extra effort. You only bother when the shrink is so severe that the careless approach produces garbage and you absolutely need the picture to stay sharp.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Quantization lowers the numeric precision of a model's weights, and sometimes its activations and KV cache, so the model needs less memory and moves fewer bytes through HBM during decode. Because LLM decode is memory bandwidth bound, fewer bytes per weight directly buys throughput and lets a model fit on cheaper hardware. A 70B model in bf16 needs about 140 GB just for weights; the same model at INT4 fits in roughly 35 GB and decodes faster because each weight read moves a quarter of the bytes. The cost is rounding error, and managing that error is what separates the two families of methods.
The interview question is not really about quantization in general. It is about a decision: given that PTQ is cheap and QAT is expensive, when is the extra cost of QAT actually justified? A weak answer reaches for QAT because it sounds more thorough. A strong answer names the single axis that matters, accuracy at a fixed bit width, and identifies the narrow regime where PTQ fails badly enough that paying for QAT is the only way to ship. Everywhere else, QAT spends real money to buy nothing the cheaper method did not already deliver.
This deep dive separates the two methods mechanically, explains why PTQ accuracy collapses rather than degrades gracefully below four bits, walks the cost asymmetry, and clears up the two distractors that trap candidates: the false belief that QAT is faster at inference, and the false belief that its calibration is cheaper. By the end you should be able to state the escalation rule in one sentence and defend it against each wrong option.
What PTQ actually does
Post-training quantization takes a fully trained, high-precision model and converts its weights to a low-bit format after the fact. There is no gradient step. The method picks scale and zero-point parameters that map the float range onto the integer grid, usually from statistics gathered over a few hundred calibration samples. Crucially, the calibration data only informs the rounding ranges; it never updates the underlying weights, which is why PTQ is so cheap.
Modern PTQ is more careful than naive rounding. GPTQ minimizes layer-wise reconstruction error using second-order information, rounding weights in an order that compensates for already-rounded ones. AWQ identifies the small fraction of salient weight channels that carry most of the activation magnitude and scales them to protect their precision. Both keep INT4 error small without touching the training pipeline. These methods exist precisely because the difference between naive and careful rounding is the difference between a usable and an unusable INT4 model.
The headline property is cost. PTQ runs in minutes on a single machine, needs no labels and no training loop, and is fully automated in serving stacks. You hand it a checkpoint and a target format and it hands back deployable weights. This is why it is the default and why the overwhelming majority of deployed quantized models are PTQ.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Dimension | PTQ | QAT |
|---|---|---|
| When applied | After training, round finished weights | During training, simulate rounding |
| Cost | Minutes, a few hundred calibration samples | Full fine-tune, gradients, GPU hours |
| Best bit width | INT8 and usually INT4 (AWQ, GPTQ) | 2-bit and 3-bit where PTQ collapses |
| What it buys | Cheap, fast, default deployment path | Accuracy recovery at aggressive low bits |
| Inference speed | Same as QAT at equal bit width | Same as PTQ at equal bit width |
Real products, models, and research that use this idea.
- Llama 4 ships official PTQ INT4 and FP8 checkpoints because the quality loss at 4-bit is negligible and no fine-tune is needed.
- AWQ and GPTQ are the de facto PTQ methods in vLLM and TensorRT-LLM for serving INT4 weights in 2026.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does PTQ accuracy collapse below 4 bits rather than degrade smoothly?
Think about how coarse the rounding grid gets at 2 or 3 bits, and that the model was trained in a smooth high-precision space with no incentive for its weights to be robust to such large rounding steps. Outlier channels dominate the error.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking QAT makes inference faster. It does not. QAT and PTQ at the same bit width run at the same speed; QAT only buys accuracy, and only where PTQ has already failed.
60 second bullets to scan on the way to the call.
Why QAT buys accuracy and never inference speed
The bit widths where PTQ stays reliable versus where it collapses
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.