Why quantization choice is an LLMOps decision, not just a serving runtime knob

Flashcard·Medium·4.0 · 0·~30s·Asked atJasper Lightning Ai Mckinsey

Attempt it

Quantization changes model behavior, not just cost and latency. INT4 typically drops quality 1-3 points on hard benchmarks, INT8 is usually near-lossless on text, FP8 (on H100 and newer) tends to be near-lossless and faster. The actual delta depends on the model family and the task.

That means a quantization change should run through the eval gate the same way a prompt change does: pick the candidate quant, run the golden set against the quantized model, compare against the FP16 baseline, gate on regression beyond threshold. Infra wants to ship INT4 for the cost win; the application team needs to confirm quality holds for their workload. Without the eval gate the team ends up shipping a behavior change blind to it.

The broader pattern: any infra decision that touches generation quality (quantization, attention backend, speculative decoding model pair) is an LLMOps decision and goes through the eval gate.

Your answer

Modern LLM serving runtimes expose a quantization knob: **FP16**, **INT8**, **INT4** (variants like AWQ and GPTQ), **FP8** (on H100 and newer NVIDIA GPUs). Each precision tier trades model quality for cost and latency. The rough quality landscape: - **FP16** is the baseline most teams compare against. - **INT8** is usually near-lossless on text tasks; quality delta within statistical noise on most benchmarks. - **FP8** (on hardware that supports it) tends to be near-lossless and runs faster than FP16. - **INT4** (AWQ, GPTQ) typically drops quality 1 to 3 points on hard benchmarks (MMLU, math, code) and may be near-lossless on easier ones. The actual delta depends on the model family (Llama vs Mistral vs Qwen) and the task (simple chat vs complex reasoning). It is not a constant. This is why quantization is an **LLMOps decision**, not just an infra knob. The infra team wants to ship the lowest precision that fits the budget, INT4 doubles requests per second per GPU. The application team needs to confirm the model still does the job at that precision. The gate that makes both sides happy is the eval CI: pick the candidate quant, run the application's golden set against the quantized model, compare against the FP16 baseline, gate on regression beyond a defined threshold. If the quant passes, ship it. If it regresses, infra picks the next-higher precision tier.

Quantization changes generation behavior, not just cost and latency. That is the property that moves it from an infra knob to an **LLMOps gate**. The 2026 quality ladder, in rough order: - **FP16/BF16**, the baseline. - **FP8** (Hopper and newer NVIDIA, MI300 on AMD), near-lossless, faster than FP16 on supported hardware. - **INT8**, near-lossless on text, modest throughput win. - **INT4** (AWQ for inference, GPTQ as a precursor format), 1 to 3 point drop on hard benchmarks, larger drop on math and code, modest drop on conversational tasks. The delta is model-family and task dependent. A Llama-3 70B quantized to INT4 may lose 2 points of MMLU; a Qwen 32B may lose 0.5; the same model on the same quant may lose less on customer-support classification than on chain-of-thought math. The operational implication: any quantization change runs through the **same eval CI gate as a prompt change**. Pick the candidate quant, run the application's golden set against the quantized deployment, compare scores against the FP16 baseline, gate on regression beyond threshold (typically 1 to 2 percent on the primary quality metric). This framing generalizes. Any infra-side decision that touches generation quality, quantization, attention backend (FlashAttention v2 vs v3), speculative decoding draft-model pairing, batch composition that might cause numerical drift, is an LLMOps decision that goes through the eval gate. The infra team owns the implementation; the LLMOps gate owns the go/no-go on behavior preservation.

TL;DR

Quantization shifts output quality, not just throughput; any change to it needs the eval gate, which makes it an LLMOps decision and not just an infra one.

Memory aid

Easy to grasp

Imagine a piano with finely-tuned strings. Picture replacing them with cheaper ones that are almost in tune but a tiny bit off. The piano still plays every song, but a trained ear notices the difference on certain pieces. Think of squeezing a number into fewer digits the same way. The cheaper strings are nearly identical on simple songs and noticeably off on hard pieces. Whether the change is okay depends on what songs the piano plays. The musician (the application team) has to listen and approve, not just the piano-tuner (the infra team) who only cares the strings cost less and the keys still go down.

Key concepts

LLMOps Inference Optimization

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Quantization gets pitched as an infra optimization: same model, fewer bits per weight, more requests per GPU, lower bill. The pitch is half-true. The throughput and cost wins are real; the implication that nothing else changes is not.

Quantization perturbs the model's numerical behavior. On benchmarks, the perturbation shows up as a one to three point score drop at INT4, often less at INT8, and sometimes near-zero at FP8. On a specific application's task distribution, the perturbation can be larger or smaller than the benchmark suggests. That uncertainty is why the decision belongs to LLMOps, not infra alone.

The 2026 quality ladder

The current quantization options for self-hosted LLM serving, ordered roughly from highest quality to lowest:

FP16/BF16. The conventional baseline. Most published benchmarks and published model weights are at this precision.

FP8. Supported in hardware on NVIDIA Hopper (H100, H200) and Blackwell (B100), and on AMD MI300 series. Quality is typically within 0.5 points of FP16 on standard benchmarks; throughput on supported hardware is meaningfully higher than FP16. The current sweet spot for teams with access to the hardware.

INT8. Near-lossless on most text tasks; modest throughput win over FP16 on most hardware. The conservative choice when FP8 hardware is not available.

INT4. Multiple methods (AWQ, GPTQ, SmoothQuant). Quality typically drops 1 to 3 points on hard benchmarks (MMLU, math, code generation), sometimes more on long-context or reasoning-heavy tasks, sometimes less on conversational tasks. The biggest throughput win, often doubles requests per second per GPU.

The ladder is not strictly monotone across all tasks. A specific (model, method, task) combination can defy expectations. Llama-3-70B at INT4-AWQ may hold MMLU within 1 point while losing 4 points on math. Qwen at INT8 may match FP16 within noise on chat but show measurable drift on structured-output tasks.

This variance is why the application's own eval set is the only reliable signal. Published benchmark numbers are inputs, not answers.

Why behavior change makes it an LLMOps decision

Method differences within a bit count

Calibration sets matter

KV cache quantization as a separate decision

The broader pattern: quality-touching infra changes need the gate

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

vLLM ships AWQ, GPTQ, FP8, and INT8 paths; SGLang and TGI cover similar ground with different default kernels.
NVIDIA TensorRT-LLM exposes hardware-optimized FP8 and INT4 paths on H100/H200 with Marlin kernels for INT4 throughput.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you decide between AWQ and GPTQ for a specific model deployment?

Run both quants through the application eval gate on the same calibration set, compare per-metric scores; AWQ tends to win on outlier-heavy distributions, GPTQ on more uniform ones.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Letting the infra team ship an INT4 quant for the cost win without running the application's eval set against it, then discovering a quality regression weeks later from customer feedback.

60 second bullets to scan on the way to the call.

The rough quality ladder: FP16, FP8, INT8, INT4
Why quantization changes behavior and not just performance

Primary sources. Browse if you want the original framing.