Where does the NVIDIA A100 sit on the inference hardware ladder?
The A100 is the 2020 Ampere-generation datacenter GPU with 40 or 80 GB HBM2e, native BF16/FP16/INT8 tensor cores, but no FP8. It is the predecessor of H100 and H200.
Think of GPU generations like car model years. The A100 is a 2020 truck: it still hauls heavy loads reliably, and many delivery fleets keep using it because it is paid off and good enough. The H100 is the 2022 model with a new fuel type the older truck cannot use, and the B200 is the 2024 model with even better mileage. Datacenter teams choose between them the way fleet managers choose between truck years: the newer ones serve heavier loads faster, but the older ones serve smaller loads cheaply. The A100 sits one rung below H100 and two rungs below B200 on this ladder, and it still does plenty of useful work for mid-size language models where the newer formats are not required.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The NVIDIA A100 is the GPU that built the 2020-2022 generation of LLM infrastructure. Most of the early Llama, Mistral, and GPT-style inference fleets ran on A100 before Hopper hardware was available at scale. In 2026 it is no longer the frontier, but it remains a workhorse for cost-sensitive serving and on-prem deployments whose capex is already sunk.
Understanding A100's place on the hardware ladder is one of those questions that looks trivial until a candidate confuses it with H100 or attributes FP8 support to it. The discriminators are precise: Ampere generation, 2020 release, HBM2e memory, BF16 and INT8 tensor cores, no native FP8. A serving engineer who knows those four facts can talk about quantization paths, fleet planning, and price-performance with confidence.
This deep dive walks the A100's specs, the lineage that places it on the ladder, the workloads that still pick it in 2026, and the workloads where it stops making sense. The goal is the mental model that lets you reason about which GPU generation fits which workload at any given price point.
Generation and release
Ampere is the GPU architecture NVIDIA introduced in 2020. The A100 is its flagship datacenter implementation, succeeding the V100 (Volta, 2017). The naming convention runs alphabetically through scientists: V is Volta, A is Ampere, H is Hopper, B is Blackwell. Each generation introduces a new SM design, new tensor-core capabilities, and usually a memory-technology bump.
A100 launched in May 2020 alongside the first PCIe 4 datacenter platforms. The SXM4 form factor delivers the full 400W TDP and NVLink 3 fabric for multi-GPU configurations; the PCIe variant is a more constrained drop-in for standard servers.
The Ampere generation also produced consumer GPUs (the RTX 30 series), which share the underlying SM design but cap tensor-core throughput and lack the HBM memory subsystem. For inference, A100 is the production-grade Ampere; the consumer cards are useful for development and very small-scale serving but not for datacenter deployment.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Many on-prem enterprise fleets bought into A100 between 2020 and 2022 and still run it in 2026 for cost-sensitive Llama-3-class workloads.
- Cloud providers (AWS, GCP, Azure, OCI) all expose A100 SKUs alongside Hopper and Blackwell because the price-per-hour gap is large enough to matter for cost-sensitive batch workloads.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does HBM bandwidth scale across A100, H100, H200, and B200?
Roughly: A100 80 GB at 2 TB/s, H100 80 GB at 3 TB/s, H200 141 GB at 4.8 TB/s, B200 192 GB at 8 TB/s. Each generation roughly doubles bandwidth, which directly raises the memory-bound decode throughput ceiling.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Claiming the A100 supports FP8 natively. FP8 tensor cores arrived with the Hopper H100 in 2022; A100 only does BF16, FP16, and INT8.
60 second bullets to scan on the way to the call.
The architecture generation and release year of A100
HBM2e capacity variants and the approximate bandwidth they deliver
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.