Compare PTQ (post-training quantization) and QAT (quantization aware training) for LLM inference. When would you choose QAT despite the extra cost? What's the ROI calculation that decides the call?
PTQ calibrates a trained model in hours and is good to about 4-bit; QAT bakes quantization into training for sub-4-bit or hard accuracy bars, at far higher cost.
Imagine compressing a high-resolution photo to save space. PTQ is like compressing the photo after it is finished: quick, and it looks fine at moderate compression, but push it too hard and edges get blocky. QAT is like the photographer knowing the photo will be heavily compressed and composing the shot to survive it: more work up front, but the result holds up under brutal compression. For most photos, fast after the fact compression is plenty. You only invest in the slow, compression-aware approach when you are squeezing extremely hard or when the photo absolutely must stay sharp. Same trade for model weights: shrink the numbers afterward when you can, train with shrinking in mind only when you must.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Quantization is the highest-leverage knob in LLM serving after the KV cache. Storing weights and activations in 4-bit or 8-bit instead of 16-bit shrinks the memory footprint, raises the batch size that fits in HBM, and accelerates the memory bandwidth bound decode phase. Because decode is bottlenecked on streaming weights and cache from HBM, halving the bytes per parameter can nearly double decode throughput. The hard part is not the arithmetic of rounding to a coarse grid. The hard part is preserving model quality while you do it.
There are two families of methods, and the interview hinges on knowing exactly where each one fits. Post-training quantization (PTQ) operates on a finished model with only a small calibration set. Quantization-aware training (QAT) folds the rounding into the training loop so the network learns to tolerate it. They sit at very different points on a cost versus quality curve, and a senior answer treats the choice as an explicit return on investment decision rather than a preference. The wrong framing is to assume QAT is simply the higher-quality option you reach for when you can afford it. The right framing is that PTQ is the default and QAT is a targeted escalation justified by specific conditions.
This deep dive defines both precisely, quantifies the gap, explains why low-bit quantization is genuinely hard, walks through the named methods you are expected to cite, and lands on the operational rule that decides the call in practice. The goal is that you can defend not just what each method is, but exactly when the extra training cost pays for itself and when it would be money set on fire.
What PTQ actually does
Post-training quantization starts from a fully trained, full-precision checkpoint and never touches the training loop. You run a small calibration set of a few hundred representative examples through the model, observe the distribution of weights and activations, and solve for the scale and zero-point that map each tensor onto the integer grid. The output is a quantized model produced in hours on a single GPU, with no labels and no gradient steps.
The naive version is round to nearest with a per-tensor scale, and it is mediocre because a single large value forces a coarse scale on everything else. The methods that matter are smarter. GPTQ performs layerwise reconstruction, adjusting the remaining weights to compensate for the error introduced by quantizing each column. AWQ identifies the small fraction of salient weight channels that most affect output and protects them with per-channel scaling. SmoothQuant addresses activations by migrating their outlier magnitude into the weights, so both sides land on grids the integer range can represent.
The headline result is that 4-bit weight-only PTQ is now routine and close to lossless on most tasks. That is precisely why PTQ is the default first move and why most production deployments never need anything more.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Dimension | PTQ | QAT |
|---|---|---|
| When applied | After training, post-hoc | During training fine-tune |
| Data needed | Small unlabeled calibration set | Labeled training data |
| Cost | Hours, single GPU | Days, multi-GPU |
| Quality drop | ~0 to 2 points | Under 0.5 point |
| Sweet spot | 4-bit weights and above | Sub-4-bit or hard accuracy bar |
| Examples | GPTQ, AWQ, SmoothQuant | Straight-through estimator fine-tune |
Real products, models, and research that use this idea.
- AWQ and GPTQ ship in vLLM and TensorRT-LLM as the default 4-bit weight PTQ paths for serving Llama 4 and Qwen 3.
- SmoothQuant is the standard recipe for W8A8 activation quantization, migrating outlier scale from activations into weights.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does PTQ tend to collapse below 4-bit while QAT survives?
Think about how the quantization error grows as the integer grid coarsens, and why calibration can only choose scales while training can reshape the weight distribution itself to fit the grid.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating QAT as the default 'better' choice. It is rarely worth its training cost; PTQ clears the bar for the large majority of deployments, so QAT is the escalation, not the starting point.
60 second bullets to scan on the way to the call.
What PTQ does and how calibration picks rounding scales
What QAT simulates during training and how gradients flow
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.