Where does the NVIDIA A100 sit on the inference hardware ladder?

Attempt it

Your answer

Quick reference

TL;DR

The A100 is the 2020 Ampere-generation datacenter GPU with 40 or 80 GB HBM2e, native BF16/FP16/INT8 tensor cores, but no FP8. It is the predecessor of H100 and H200.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of GPU generations like car model years. The A100 is a 2020 truck: it still hauls heavy loads reliably, and many delivery fleets keep using it because it is paid off and good enough. The H100 is the 2022 model with a new fuel type the older truck cannot use, and the B200 is the 2024 model with even better mileage. Datacenter teams choose between them the way fleet managers choose between truck years: the newer ones serve heavier loads faster, but the older ones serve smaller loads cheaply. The A100 sits one rung below H100 and two rungs below B200 on this ladder, and it still does plenty of useful work for mid-size language models where the newer formats are not required.

Key concepts

Inference hardware basics NVIDIA A100 GPU

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

2 min: A100 generation and release year + HBM2e capacity and bandwidth + supported tensor-core formats and the FP8 gap + generation ladder + 2026 production use cases.

Real-world examples

Real products, models, and research that use this idea.

Many on-prem enterprise fleets bought into A100 between 2020 and 2022 and still run it in 2026 for cost-sensitive Llama-3-class workloads.
Cloud providers (AWS, GCP, Azure, OCI) all expose A100 SKUs alongside Hopper and Blackwell because the price-per-hour gap is large enough to matter for cost-sensitive batch workloads.
Lambda Labs and CoreWeave list A100 80 GB at a significant discount versus H100, which is why startup-scale inference often still lands there.
vLLM supports A100 as a first-class target alongside H100 and B200 because the install base is too large to deprecate.
Hugging Face's TGI documents A100 as a fully supported deployment target with INT8 quantization for models up to 70B in the 80 GB variant.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does HBM bandwidth scale across A100, H100, H200, and B200?

Sign in or go Pro to see the approach for this follow-up.

QWhen does INT8 on A100 still beat BF16 on H100 economically?

Sign in or go Pro to see the approach for this follow-up.

QWhy does the 40 GB A100 see less production use than the 80 GB SKU?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming the A100 supports FP8 natively. FP8 tensor cores arrived with the Hopper H100 in 2022; A100 only does BF16, FP16, and INT8.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.