Compare PTQ (post-training quantization) and QAT (quantization aware training) for LLM inference. When would you choose QAT despite the extra cost? What's the ROI calculation that decides the call?
PTQ calibrates a trained model in hours and is good to about 4-bit; QAT builds quantization into training for sub-4-bit or hard accuracy bars, at far higher cost.
Imagine compressing a high-resolution photo to save space. PTQ is like compressing the photo after it is finished: quick, and it looks fine at moderate compression, but push it too hard and edges get blocky. QAT is like the photographer knowing the photo will be heavily compressed and composing the shot to survive it: more work up front, but the result holds up under heavy compression. For most photos, fast after-the-fact compression is plenty. You only invest in the slow, compression-aware approach when you are squeezing extremely hard or when the photo must stay sharp. Same trade for model weights: shrink the numbers afterward when you can, train with shrinking in mind only when you must.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Quantization is the largest efficiency gain in LLM serving after KV cache management. Storing weights and activations in 4-bit or 8-bit instead of 16-bit shrinks the memory footprint, raises the batch size that fits in HBM, and accelerates the memory bandwidth bound decode phase. Because decode is bottlenecked on streaming weights and cache from HBM, halving the bytes per parameter can nearly double decode throughput. The hard part is not the arithmetic of rounding to a coarse grid. The hard part is preserving model quality while you do it.
There are two families of methods, and each fits a different case. Post-training quantization (PTQ) operates on a finished model with only a small calibration set. Quantization-aware training (QAT) folds the rounding into the training loop so the network learns to tolerate it. They sit at very different points on a cost versus quality curve, so the choice is an explicit return-on-investment decision rather than a preference. It is wrong to assume QAT is the higher-quality option to use whenever you can afford it. PTQ is the default, and QAT is a targeted escalation justified by specific conditions.
What PTQ actually does
Post-training quantization starts from a fully trained, full-precision checkpoint and never touches the training loop. You run a small calibration set of a few hundred representative examples through the model, observe the distribution of weights and activations, and solve for the scale and zero-point that map each tensor onto the integer grid. The output is a quantized model produced in hours on a single GPU, with no labels and no gradient steps.
The naive version is round to nearest with a per-tensor scale, and it is weak because a single large value forces a coarse scale on everything else. Better methods do more. GPTQ performs layerwise reconstruction, adjusting the remaining weights to compensate for the error introduced by quantizing each column. AWQ identifies the small fraction of salient weight channels that most affect output and protects them with per-channel scaling. SmoothQuant addresses activations by migrating their outlier magnitude into the weights, so both sides land on grids the integer range can represent.
The main result is that 4-bit weight-only PTQ is routine and close to lossless on most tasks. That is why PTQ is the default first move and why most production deployments never need anything more.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Dimension | PTQ | QAT |
|---|---|---|
| When applied | After training, post-hoc | During a training or fine-tuning run |
| Data needed | Small unlabeled calibration set | Labeled training data |
| Cost | Hours, single GPU | Days, multi-GPU |
| Quality drop | ~0 to 2 points | Under 0.5 point |
| Sweet spot | 4-bit weights and above | Sub-4-bit or hard accuracy bar |
| Examples | GPTQ, AWQ, SmoothQuant | Straight-through estimator fine-tune |
Real products, models, and research that use this idea.
- AWQ and GPTQ ship in vLLM and TensorRT-LLM as the standard 4-bit weight PTQ paths for serving Llama 4 and Qwen 3.
- SmoothQuant is a standard training-free W8A8 method, migrating outlier scale from activations into weights.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does PTQ tend to collapse below 4-bit while QAT survives?
Think about how the quantization error grows as the integer grid coarsens, and why calibration can only choose scales while training can reshape the weight distribution itself to fit the grid.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating QAT as the default 'better' choice. It is rarely worth its training cost; PTQ clears the bar for the large majority of deployments, so QAT is the escalation, not the starting point.
60 second bullets to scan on the way to the call.
What PTQ does and how calibration picks rounding scales
What QAT simulates during training and how gradients flow
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.