When would you choose QAT over PTQ for inference quantization and what's the cost?
Compare PTQ (post-training quantization) and QAT (quantization-aware training) for LLM inference. When would you choose QAT despite the extra cost? What's the ROI calculation that decides the call?
PTQ calibrates a trained model in hours and is good to about 4-bit; QAT bakes quantization into training for sub-4-bit or hard accuracy bars, at far higher cost.
Imagine compressing a high-resolution photo to save space. PTQ is like compressing the photo after it is finished: quick, and it looks fine at moderate compression, but push it too hard and edges get blocky. QAT is like the photographer knowing the photo will be heavily compressed and composing the shot to survive it: more work up front, but the result holds up under brutal compression. For most photos, fast after the fact compression is plenty. You only invest in the slow, compression-aware approach when you are squeezing extremely hard or when the photo absolutely must stay sharp. Same trade for model weights: shrink the numbers afterward when you can, train with shrinking in mind only when you must.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: define PTQ versus QAT + quality and cost gap + named methods (GPTQ, AWQ, SmoothQuant) + activation outliers + sub-4-bit escalation + the start-PTQ-escalate on failure rule.
| Dimension | PTQ | QAT |
|---|---|---|
| When applied | After training, post-hoc | During training fine-tune |
| Data needed | Small unlabeled calibration set | Labeled training data |
| Cost | Hours, single GPU | Days, multi-GPU |
| Quality drop | ~0 to 2 points | Under 0.5 point |
| Sweet spot | 4-bit weights and above | Sub-4-bit or hard accuracy bar |
| Examples | GPTQ, AWQ, SmoothQuant | Straight-through estimator fine-tune |
Real products, models, and research that use this idea.
- AWQ and GPTQ ship in vLLM and TensorRT-LLM as the default 4-bit weight PTQ paths for serving Llama 4 and Qwen 3.
- SmoothQuant is the standard recipe for W8A8 activation quantization, migrating outlier scale from activations into weights.
- DeepSeek V4 and Llama 4 are commonly served with FP8 weights and FP8 KV cache as a PTQ-grade default on H100 and B200 hardware.
- Llama-family QAT checkpoints released by Meta target 4-bit on-device deployment where PTQ accuracy was insufficient for the quality bar.
- NVIDIA TensorRT Model Optimizer exposes both PTQ calibration and QAT fine-tuning behind one toolkit for production quantization.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does PTQ tend to collapse below 4-bit while QAT survives?
QWhat makes activation quantization harder than weight quantization?
QHow does KV-cache quantization differ from quantizing the weights?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating QAT as the default 'better' choice. It is rarely worth its training cost; PTQ clears the bar for the large majority of deployments, so QAT is the escalation, not the starting point.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.