FLOPs vs FLOP/s: fill in the counts versus rate distinction.

Attempt it

FLOPs (lowercase 's') is a of floating-point operations, a quantity with no time component. FLOP/s is a , measured in operations per second. The prefill phase of LLM inference does many FLOPs and tends to be -bound, while single-token decode does very few FLOPs per byte read from HBM and is -bound.

Quick reference

TL;DR

FLOPs is a count of operations; FLOP/s is a rate. Prefill does many FLOPs and is compute-bound, while decode does few FLOPs per byte read and is bandwidth-bound.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine measuring how much running someone does. You can count total steps taken, a pure number with no clock attached. Or you can measure steps per second, which only makes sense alongside a stopwatch. FLOPs and FLOP/s are exactly that pair. FLOPs is the count of math operations a job needs, like asking how many steps fit in a marathon. FLOP/s is how fast a chip can do math, like asking how fast a runner moves. The interesting question is whether a chip spends its time waiting on data or actually doing math. When the prompt is processed all at once, the chip is busy computing. When the model produces output one word at a time, the chip is mostly waiting for data from memory.

Key concepts

FLOPs vs FLOP/s Arithmetic intensity Prefill phase Memory-bound vs compute-bound

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: nail the count versus rate distinction, give the 2NT forward-pass estimate, anchor peak FLOP/s for H100 and B200 across dtypes, place prefill and decode on the roofline using arithmetic intensity, then close with how to use the pair to localise bottlenecks.

Real-world examples

Real products, models, and research that use this idea.

NVIDIA H100 SXM peak is roughly 990 TFLOP/s BF16 and 1980 TFLOP/s FP8, the headline numbers in every Hopper spec sheet.
Llama 4 Maverick forward pass at 4k context costs roughly 4 to 5 PFLOPs total, the figure used by serving teams to size GPU pools.
DeepSeek V4 published training FLOPs estimates around 6e24, comparable to recent frontier models, justifying the use of large H200 clusters.
B200 Blackwell GPUs roughly double H100 FP8 FLOP/s, which is why frontier labs migrated training and serving to them through 2025.
Roofline analyses for vLLM and TensorRT-LLM publish achieved FLOP/s curves against batch size, showing decode climbing from below 1 percent of peak toward saturation.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the transformer forward pass FLOPs estimate 2NT and not just NT?

Sign in or go Pro to see the approach for this follow-up.

QWhy does FP8 nearly double the H100 peak FLOP/s versus BF16?

Sign in or go Pro to see the approach for this follow-up.

QHow would you estimate FLOPs for an attention layer specifically?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating FLOPs and FLOP/s as synonyms. The first is a job size; the second is a hardware throughput rate. Confusing them produces nonsense like saying an H100 has 990 trillion FLOPs.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.