Identify the NVIDIA H100: what generation, what memory, what tensor-core formats?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

The H100 is NVIDIA's Hopper-generation data-center GPU (2022-2024 design, still heavily deployed in 2026). It ships with 80 GB of HBM3 at roughly 3.35 TB/s bandwidth and adds native FP8 tensor cores (E4M3 / E5M2) alongside the existing BF16 / FP16 / TF32 / INT8 paths. It has been the workhorse for LLM serving from 2023 onward, now sharing duty with H200 and the newer Blackwell B100/B200.

Your answer

The **NVIDIA H100** is the data-center GPU that powered the explosion of LLM serving from 2023 onward. It belongs to NVIDIA's Hopper architecture generation, which followed Ampere (A100) and preceded Blackwell (B100, B200). Three facts anchor it. First, it ships with **80 GB of HBM3 memory** at roughly 3.35 TB/s of bandwidth. Both numbers are meaningful: 80 GB fits a 70B model in FP8 with room for batch and KV cache; 3.35 TB/s sets the ceiling on how fast decode can read weights each step. Second, it adds **native FP8 tensor cores** to the tensor-core formats inherited from Ampere. FP8 doubles matmul throughput versus FP16 on the same silicon and is the single biggest reason serving moved from BF16 to FP8 starting in 2023. Third, it provides about 990 TFLOP/s of BF16 throughput and roughly 1980 TFLOP/s in FP8, peak rates that anchor every roofline calculation in the inference-optimization topic. The H100 launched in 2022 and is still the workhorse in 2026, now shipping alongside the H200 (more memory, same compute) and the newer Blackwell B100/B200 (roughly 2x throughput, native FP4).

**The H100** is NVIDIA's flagship Hopper-generation data-center GPU, launched in 2022. It defined the hardware substrate for production LLM serving from 2023 onward and remains heavily deployed in 2026. Key specs to know cold: - **HBM3** memory at 80 GB capacity, with about 3.35 TB/s of bandwidth on the SXM variant. The PCIe variant uses HBM3 at slightly lower bandwidth. - **Tensor cores** supporting BF16, FP16, TF32, INT8, and (newly with Hopper) FP8 in both E4M3 and E5M2 formats. - **Peak throughput** around 990 TFLOP/s BF16, 1980 TFLOP/s FP8 with sparsity off, both dense matmul rates. - **Transformer Engine**: a software/hardware combination that automates FP8 mixed precision for transformer layers. - **NVLink** at 900 GB/s for inter-GPU communication, with up to 8 GPUs in an HGX system. The two changes that mattered most for LLM serving versus the predecessor A100 were native FP8 tensor cores and the bandwidth jump from 2.0 TB/s (A100) to 3.35 TB/s (H100 SXM). FP8 halves bytes-per-token and roughly doubles matmul throughput. Higher bandwidth eases the bandwidth-bound decode bottleneck. In 2026 the H100 is no longer the newest part. H200 keeps the same compute but bumps HBM to 141 GB at 4.8 TB/s. Blackwell B100/B200 roughly doubles FP8 throughput and adds native FP4. But the installed base of H100 is enormous, so it remains the reference platform for most serving discussions.

Quick reference

TL;DR

The H100 is NVIDIA's Hopper-generation data-center GPU, 80 GB HBM3 at 3.35 TB/s, with native FP8 tensor cores; it has been the LLM serving workhorse since 2023.

Memory aid

Easy to grasp

Picture the standard kitchen most professional chefs cooked in for the last few years. It has a fridge of a certain size, a counter that can hold a certain amount of work, and a stove that can run at a specific top speed. The H100 is that kitchen for LLM serving. It carries 80 gigabytes of fast memory, can move data in and out of that memory at roughly 3.35 terabytes per second, and has special-purpose math units that can chew through numbers at extraordinary speed, especially at a new tiny number format that earlier kitchens did not understand. It launched in 2022, replaced the older A100 across most LLM workloads, and remains everywhere in 2026 even though newer kitchens have started arriving.

Key concepts

NVIDIA H100 Hopper architecture HBM (high-bandwidth memory)Tensor cores

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: anchor H100 as Hopper-generation with 80 GB HBM3 at 3.35 TB/s, walk through native FP8 tensor cores and peak rates, compare against A100 and the newer H200 and B200, then close on the Transformer Engine and operational role of H100 in 2026 serving.

Real-world examples

Real products, models, and research that use this idea.

OpenAI's GPT-5.5 serving fleet runs primarily on H100 and H200 clusters with FP8 quantisation.
Anthropic deploys Claude Opus 4.7 across multi-node H100 SXM nodes with NVLink-coupled tensor parallel.
Together AI and Modal price H100 SXM at about 3 to 4 dollars per GPU-hour for on-demand serving as of 2026.
Meta's Llama 3.1 405B reference inference recipes target 8x H100 with FP8 weights and BF16 attention.
DeepSeek V4's training run famously used H800 GPUs (China-export-compliant H100 variant) at large scale.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy did the H100 push so much harder on FP8 than the A100 did on INT8?

QHow does the H200 differ from the H100 for LLM serving in practice?

QWhat does Hopper's MIG (Multi-Instance GPU) feature do for inference serving?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating H100 as just a faster A100. The big architectural change is native FP8 tensor cores and roughly 2x the HBM bandwidth, both of which reshape how LLM serving sizes batch and quantises weights.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.