Compare PTQ (post-training quantization) and QAT (quantization-aware training) for LLM inference. When would you choose QAT despite the extra cost? What's the ROI calculation that decides the call?
Free · 2 AI evals / day
**Quantization** shrinks a model by storing its weights in fewer bits, say 4-bit integers instead of 16-bit floats. That cuts memory and speeds up the memory-bound decode phase. The question is how you pick the low-bit values without wrecking quality.
Post-training quantization, or PTQ, does it after the model is fully trained. You feed in a small calibration set of a few hundred examples, measure the range of each weight and activation, and pick rounding scales. It takes hours on one GPU and no labels. Methods like GPTQ, AWQ, and SmoothQuant are all PTQ.
Quantization-aware training, or QAT, instead simulates the rounding during training. The forward pass rounds weights to low bits, and gradients still flow through so the network learns weights that tolerate rounding. This costs a full fine-tune: more GPUs, more time, and usually labeled data.
The practical rule is simple. Start with PTQ because it is cheap and usually good enough to about 4-bit. Reach for QAT only when PTQ misses a hard accuracy bar or you are pushing to very low bit widths like 2-bit or 3-bit, where PTQ tends to collapse.
**PTQ** quantizes an already-trained model. You pass a small calibration set, measure activation and weight ranges, and solve for rounding scales. GPTQ, AWQ, and SmoothQuant are PTQ. It runs in hours on one GPU with no labels, and lands within roughly 0 to 2 points of full precision down to about 4-bit weights.
QAT instead injects rounding noise into the forward pass during training. Gradients pass through a straight-through estimator, so the network learns weights robust to quantization. That shrinks the gap to under half a point at the same bit width. The price is a full fine-tune: multi-GPU time, training data, and pipeline complexity measured in days rather than hours.
The decision is an ROI question. Choose QAT only when PTQ misses a product-critical accuracy bar. Also choose it for extreme low-bit targets such as 2-bit or 3-bit, where PTQ collapses under activation outliers. A third case is when you control training and amortize the cost across many deployments.
For almost everyone the rule holds. Start with PTQ, validate on real task evals, and escalate to QAT only on failure.
Quick reference
TL;DR
PTQ calibrates a trained model in hours and is good to about 4-bit; QAT bakes quantization into training for sub-4-bit or hard accuracy bars, at far higher cost.
Imagine compressing a high-resolution photo to save space. PTQ is like compressing the photo after it is finished: quick, and it looks fine at moderate compression, but push it too hard and edges get blocky. QAT is like the photographer knowing the photo will be heavily compressed and composing the shot to survive it: more work up front, but the result holds up under brutal compression. For most photos, fast after the fact compression is plenty. You only invest in the slow, compression-aware approach when you are squeezing extremely hard or when the photo absolutely must stay sharp. Same trade for model weights: shrink the numbers afterward when you can, train with shrinking in mind only when you must.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Reaching for QAT before PTQ has even been tried on the model PTQ is far cheaper and clears the accuracy bar for most deployments, so it should always be the first attempt.
Choosing QAT at 4-bit or 8-bit weight-only targets Modern PTQ methods are near lossless in that range, so the training cost buys almost nothing.
Quantizing weights when the model is already memory-bound on the KV cache Weight quantization barely helps if the cache dominates HBM, so quantize the cache instead.
Applying either method without a real downstream evaluation harness Perplexity hides task regressions, so quantizing blind can ship a model that silently fails on real workloads.
Quick scan · 2–4 min
Everything important, quickly.
4 min: define PTQ versus QAT + quality and cost gap + named methods (GPTQ, AWQ, SmoothQuant) + activation outliers + sub-4-bit escalation + the start-PTQ-escalate on failure rule.
Side-by-side
Dimension
PTQ
QAT
When applied
After training, post-hoc
During training fine-tune
Data needed
Small unlabeled calibration set
Labeled training data
Cost
Hours, single GPU
Days, multi-GPU
Quality drop
~0 to 2 points
Under 0.5 point
Sweet spot
4-bit weights and above
Sub-4-bit or hard accuracy bar
Examples
GPTQ, AWQ, SmoothQuant
Straight-through estimator fine-tune
Real-world examples
Real products, models, and research that use this idea.
1AWQ and GPTQ ship in vLLM and TensorRT-LLM as the default 4-bit weight PTQ paths for serving Llama 4 and Qwen 3.
2SmoothQuant is the standard recipe for W8A8 activation quantization, migrating outlier scale from activations into weights.
3DeepSeek V4 and Llama 4 are commonly served with FP8 weights and FP8 KV cache as a PTQ-grade default on H100 and B200 hardware.
4Llama-family QAT checkpoints released by Meta target 4-bit on-device deployment where PTQ accuracy was insufficient for the quality bar.
5NVIDIA TensorRT Model Optimizer exposes both PTQ calibration and QAT fine-tuning behind one toolkit for production quantization.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does PTQ tend to collapse below 4-bit while QAT survives?
Sign in or go Pro to see the approach for this follow-up.
QWhat makes activation quantization harder than weight quantization?
Sign in or go Pro to see the approach for this follow-up.
QHow does KV-cache quantization differ from quantizing the weights?
Sign in or go Pro to see the approach for this follow-up.
QHow would you decide the ROI of QAT for a specific deployment?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating QAT as the default 'better' choice. It is rarely worth its training cost; PTQ clears the bar for the large majority of deployments, so QAT is the escalation, not the starting point.