FLOPs vs FLOP/s: fill in the counts versus rate distinction.
FLOPs is a count of operations; FLOP/s is a rate. Prefill does many FLOPs and is compute-bound, while decode does few FLOPs per byte read and is bandwidth-bound.
Imagine measuring how much running someone does. You can count total steps taken, a pure number with no clock attached. Or you can measure steps per second, which only makes sense alongside a stopwatch. FLOPs and FLOP/s are exactly that pair. FLOPs is the count of math operations a job needs, like asking how many steps fit in a marathon. FLOP/s is how fast a chip can do math, like asking how fast a runner moves. The interesting question is whether a chip spends its time waiting on data or actually doing math. When the prompt is processed all at once, the chip is busy computing. When the model produces output one word at a time, the chip is mostly waiting for data from memory.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
FLOPs versus FLOP/s is the most commonly confused unit pair in ML. People conflate the two constantly, and the conflation hides the real bottleneck of every kernel they reason about. Get the distinction clean and a huge amount of inference performance discussion suddenly makes sense.
This deep dive defines each term precisely, derives the standard transformer forward-pass FLOPs estimate, anchors current hardware peak rates for H100 and B200 across dtypes, walks through arithmetic intensity and the roofline model, and finishes with the prefill versus decode split that the rest of the inference-optimization topic builds on.
By the end you should be able to read a model card and a GPU spec sheet together, compute a theoretical compute time, recognise immediately whether bandwidth or compute will dominate in practice, and explain to a colleague why a kernel that should take a millisecond actually takes forty. Those mental moves are the foundation of everything else in LLM serving.
FLOPs is a count, FLOP/s is a rate
FLOPs with a lowercase trailing s is the plural of FLOP, a floating-point operation. It is a count, dimensionally an integer. A single multiply or add is one FLOP. A multiply-accumulate counts as two FLOPs. Job sizes are expressed in FLOPs, never per second.
FLOP/s with the slash is operations per second. Dimensionally it is operations over time, the inverse of how long each operation takes. Hardware throughput is expressed in FLOP/s, with prefixes mega (1e6), giga (1e9), tera (1e12), peta (1e15), exa (1e18).
The unfortunate convention is that the lowercase s in FLOPs (count) looks similar to the s in FLOP/s (rate). Many sources drop the slash and write FLOPS as the rate, making the ambiguity worse. The clean way to disambiguate in writing is to always put the slash for the rate, FLOP/s, and trust the lowercase s as the plural marker for the count. Some authors use FLOPs for the count and FLOPS (all caps) for the rate; pick a convention and stick to it.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- NVIDIA H100 SXM peak is roughly 990 TFLOP/s BF16 and 1980 TFLOP/s FP8, the headline numbers in every Hopper spec sheet.
- Llama 4 Maverick forward pass at 4k context costs roughly 4 to 5 PFLOPs total, the figure used by serving teams to size GPU pools.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the transformer forward pass FLOPs estimate 2NT and not just NT?
Each multiply-accumulate counts as two operations: one multiply, one add. With N parameters each participating in one MAC per token, you get 2 FLOPs per parameter per token, hence 2NT total. The factor of 2 also captures the standard FMA convention used in all hardware peak numbers.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating FLOPs and FLOP/s as synonyms. The first is a job size; the second is a hardware throughput rate. Confusing them produces nonsense like saying an H100 has 990 trillion FLOPs.
60 second bullets to scan on the way to the call.
The definition of FLOPs as a count of floating-point operations
The definition of FLOP/s as a rate, operations per second
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.