Explain what NVIDIA CAGRA buys you over CPU HNSW and the operational catch that decides whether you should reach for it.

Flashcard·Medium·4.0 · 0·~30s·Asked atZilliz·Relevant atNVIDIA

Attempt it

RAFT and CAGRA are NVIDIA's GPU primitives for ANN. CAGRA is a graph based index conceptually similar to HNSW but designed for GPU parallel traversal; RAFT is the wider numerical primitives library that CAGRA sits inside.

The win: roughly 5-10x QPS over CPU HNSW at the same recall, especially on high-dim vectors where per-distance dot product cost dominates. At billion-vector scale on a multi-GPU server, the gap can widen to 10-20x.

The operational catch is twofold. (1) GPU HBM is smaller and more expensive than CPU RAM, an 80GB H100 holds far fewer vectors than a 1TB CPU box, so very large indexes either don't fit or require multiple GPUs and inter-GPU sharding. (2) GPU instances cost 5-10x what CPU instances cost per byte of memory, so GPU-ANN makes sense when QPS or per query latency is the binding constraint, not when storage cost is.

Milvus and Vespa ship CAGRA integration in 2026. It is the natural choice for very high QPS retrieval, for workloads where the embedding side already lives on GPU (so the vectors never leave HBM), or for latency critical RAG over mid scale corpora.

Your answer

**CAGRA** (CUDA Accelerated Graph Index) is NVIDIA's GPU-native ANN algorithm, shipping inside the **RAFT** library of GPU primitives. Conceptually it is a graph based ANN like HNSW: each vector is a node, edges connect near neighbors, search is a greedy walk. The difference is that the graph and the distance computations all live in GPU HBM (the GPU's on-chip memory), and the search exploits GPU parallelism heavily. **What it buys you.** Per-query throughput is roughly 5-10x higher than the best CPU HNSW at the same recall, on the same hardware budget. The win is largest when: - The vectors are high-dim (1024+), so per-distance computation cost dominates. - The QPS is high enough that GPU batching helps (CAGRA processes many queries in parallel). - The workload is read heavy with a stable index (CAGRA's build is less incremental friendly than HNSW). **The catches.** Two main ones. *GPU memory is small.* An H100 has 80GB of HBM. A billion 1024-dim float32 vectors is 4TB, so the index does not fit on one GPU; you either shard across many GPUs (operationally complex) or use compression (PQ on GPU). CPU boxes routinely have 256GB to 1TB of RAM at a fraction of the cost. *GPU instances are expensive per byte.* An A100/H100 cloud instance costs 5-10x what a comparable CPU instance costs per byte of RAM. So if the binding constraint on your bill is storage cost (which it usually is at billion-vector scale), GPU-ANN makes it worse, not better. **When to reach for it.** Milvus and Vespa both ship CAGRA integration in 2026. Pick it when your binding constraint is QPS or per query latency, not when it is storage cost. Natural fits: high QPS retrieval backends, workloads where embeddings are already generated on GPU and stay in HBM, latency critical RAG over mid scale corpora (1M-100M vectors).

**CAGRA and the RAFT library.** RAFT (Rapids Accelerated Framework Toolkit) is NVIDIA's library of GPU numerical primitives, of which CAGRA (CUDA Accelerated Graph Index) is the ANN component. CAGRA builds a small world graph similar in spirit to HNSW but with a flatter structure designed for the GPU's parallel execution model. Both the graph and the vectors live in GPU HBM during serving; the search kernel exploits the GPU's thousands of cores to evaluate many distance computations in parallel. **The throughput win.** At the same recall (typically measured at recall@10 = 0.95), CAGRA delivers 5-10x higher QPS than the best tuned CPU HNSW on the same hardware budget. The win compounds with vector dimensionality: at 1024 dims the gap is 5x, at 4096 dims it widens to 10-15x because the per-distance cost dominates and GPUs are dramatically better at fused multiply-adds. Per-query latency is also lower (often sub-millisecond at small QPS), but the bigger win is batched QPS where the GPU's parallelism shines. **The operational catches in detail.** 1. **GPU HBM is small and expensive.** An H100 has 80GB of HBM; an H200 has 141GB; an MI300X has 192GB. The largest single-GPU box is bounded by these numbers. A 1B-vector index at 1024 dims float32 is 4TB raw, which requires 25-50 GPUs to hold even with aggressive sharding. CPU boxes with 1TB of RAM are routine and dramatically cheaper per byte. 2. **Cost-per-vector inversion.** Cloud GPU instances cost 5-10x what CPU instances cost per byte of memory. For storage bound workloads, GPU-ANN is the worse economic choice. 3. **Less incremental friendly.** CAGRA's build is faster than CPU HNSW but the incremental-insert story is weaker. Workloads with continuous high rate writes still prefer CPU HNSW or hybrid approaches. 4. **Index portability.** A CAGRA built index is not directly compatible with CPU only deployments; switching off GPU means rebuilding. **The right workloads.** - **High-QPS retrieval.** Public-facing

Quick reference

TL;DR

CAGRA (NVIDIA's GPU graph-ANN, inside the RAFT library) gives you roughly 5-10x QPS over CPU HNSW at the same recall.

Memory aid

Easy to grasp

A CPU is like one really smart accountant doing additions one at a time. A GPU is like a stadium full of slightly dumber accountants who can all add at the same time. For vector search, where you need to compute thousands of small dot products, the stadium wins by a huge margin, often 10x faster. **But** the stadium has a tiny private vault, way smaller than the accountant's filing cabinet, and renting the stadium is much more expensive per square foot. So you bring the stadium in when you have a lot of queries pouring in and need them answered now; you don't bring it in just to store a billion items cheaply.

Key concepts

CAGRA

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: name RAFT/CAGRA + the throughput win + HBM and cost catches + Milvus/Vespa integration + the decision rubric (QPS-binding vs cost-binding).

Side-by-side

Property	CPU HNSW	CAGRA (GPU)	DiskANN
Primary storage	RAM	GPU HBM	SSD
Per-query latency	~1-3 ms	~0.2-1 ms	~5-15 ms
Sustained QPS	100-1000/core	10k-100k/GPU	100-500/box
$/byte of storage	1x baseline	5-10x worse	0.1-0.3x baseline
Max single-node index	~1TB RAM	80-192GB HBM	Multi-TB SSD
Best for	Default production	High QPS, low latency, mid-scale	Cost first billion scale

Real-world examples

Real products, models, and research that use this idea.

Milvus 2.4+ ships CAGRA as a first class index type, recommended for high-QPS RAG and recommendation backends.
Vespa offers CAGRA integration via its tensor framework for hybrid scoring at scale.
NVIDIA cuVS is the standalone library teams use to wire RAFT/CAGRA into custom serving stacks.
Production vLLM + CAGRA deployments at frontier AI labs keep embedding and retrieval co-located on the same GPU, eliminating CPU-GPU transfer.
BAAI and Anthropic published 2024-2025 benchmarks showing CAGRA reaching 5-10x QPS over CPU HNSW on 10M-100M-vector benchmarks at the same recall.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does CAGRA pair with PQ to handle indexes that exceed HBM?

QWhy is the GPU embedding co-location story the killer use case in 2026?

QWhat is the rough crossover QPS where CAGRA beats CPU HNSW on $/query?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming GPU-ANN is universally better than CPU HNSW because GPUs are faster. The cost per byte of memory inversion means GPU-ANN is the wrong choice for cost first storage bound workloads; it shines only when QPS or latency is the binding constraint.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.