Identify the NVIDIA H100: what generation, what memory, what tensor core formats?
The H100 is NVIDIA's Hopper-generation data-center GPU, 80 GB HBM3 at 3.35 TB/s, with native FP8 tensor cores; it has been the LLM serving workhorse since 2023.
Picture the standard kitchen most professional chefs cooked in for the last few years. It has a fridge of a certain size, a counter that can hold a certain amount of work, and a stove that can run at a specific top speed. The H100 is that kitchen for LLM serving. It carries 80 gigabytes of fast memory, can move data in and out of that memory at roughly 3.35 terabytes per second, and has special-purpose math units that can chew through numbers at extraordinary speed, especially at a new tiny number format that earlier kitchens did not understand. It launched in 2022, replaced the older A100 across most LLM workloads, and remains everywhere in 2026 even though newer kitchens have started arriving.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The NVIDIA H100 is the hardware backbone of LLM serving in the modern era. From its 2022 launch through 2026 it has been the most-deployed GPU for production inference of large models, and its specs anchor every roofline, every batch-size calculation, every cost estimate in the inference-optimization topic.
Knowing the H100 well is not about memorising marketing numbers. It is about being able to walk into a serving design review and have informed opinions on how big a model fits per GPU, how fast each decode step runs, where the bandwidth ceiling sits, and how the architecture differs from the older A100 and newer Blackwell parts.
This deep dive covers the H100's architecture generation (Hopper), its memory subsystem (HBM3, 80 GB, 3.35 TB/s), the tensor-core formats it accelerates natively (the headline addition being FP8), the supporting software (Transformer Engine), the variants (SXM versus PCIe, H100 versus H200), and its place in the lineage from A100 to B200.
By the end you should be able to answer questions like: why did production serving move from BF16 to FP8 starting in 2023? Why is a 70B model in FP8 the canonical single-H100 workload? When does it make sense to migrate to H200 or B200? And what does the Transformer Engine actually do under the hood?
The Hopper architecture in context
NVIDIA's data-center GPU lineage runs through named architecture generations: Pascal (P100, 2016), Volta (V100, 2017), Ampere (A100, 2020), Hopper (H100, 2022), Blackwell (B100/B200, 2024). Each generation roughly doubles peak compute and adds new tensor-core formats.
Hopper's specific innovations for LLMs were threefold. First, native FP8 tensor cores, the headline format change. Second, the Transformer Engine, a hardware-software pair that automates FP8 mixed precision for transformer blocks. Third, a substantial bandwidth bump from HBM2e on A100 (1.55-2.0 TB/s) to HBM3 on H100 (3.35 TB/s on SXM).
Hopper also introduced thread block clusters, distributed shared memory, and DPX instructions for dynamic programming, but those are less directly relevant to LLM serving than FP8 and bandwidth. The H100 launched into the early ChatGPT era and immediately became the default substrate for serving large transformer models.
In 2026 Hopper is no longer the newest generation; Blackwell (B100, B200, GB200) is shipping in volume. But the deployed base of H100 across hyperscalers, neoclouds, and enterprise data centres is enormous, so it remains the most discussed GPU in serving conversations.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's GPT-5.5 serving fleet runs primarily on H100 and H200 clusters with FP8 quantisation.
- Anthropic deploys Claude Opus 4.7 across multi-node H100 SXM nodes with NVLink-coupled tensor parallel.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did the H100 push so much harder on FP8 than the A100 did on INT8?
INT8 has limited dynamic range and requires aggressive per-tensor scaling that breaks down on outlier activations common in LLMs. FP8 (especially E4M3) preserves enough range to handle outliers with simpler calibration, making it deployable for LLM inference at production accuracy. Hopper's Transformer Engine made FP8 mixed precision automatic, which INT8 never achieved.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating H100 as just a faster A100. The big architectural change is native FP8 tensor cores and roughly 2x the HBM bandwidth, both of which reshape how LLM serving sizes batch and quantises weights.
60 second bullets to scan on the way to the call.
The architecture generation H100 belongs to (Hopper)
Its HBM capacity (80 GB) and bandwidth (3.35 TB/s on SXM)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.