Explain what NVIDIA CAGRA buys you over CPU HNSW and the operational catch that decides whether you should reach for it.

Flashcard·Medium·4.0 · 0·~30s·Asked atForethought Gnani Zilliz·Relevant atNVIDIA

Attempt it

RAFT and CAGRA are NVIDIA's GPU primitives for ANN. CAGRA is a graph based index conceptually similar to HNSW but designed for GPU parallel traversal; RAFT is the wider numerical primitives library that CAGRA sits inside.

The win: roughly 5-10x QPS over CPU HNSW at the same recall, especially on high-dim vectors where per-distance dot product cost dominates. At billion-vector scale on a multi-GPU server, the gap can widen to 10-20x.

The operational catch is twofold. (1) GPU HBM is smaller and more expensive than CPU RAM, an 80GB H100 holds far fewer vectors than a 1TB CPU box, so very large indexes either don't fit or require multiple GPUs and inter-GPU sharding. (2) GPU instances cost 5-10x what CPU instances cost per byte of memory, so GPU-ANN makes sense when QPS or per query latency is the binding constraint, not when storage cost is.

Milvus and Vespa ship CAGRA integration in 2026. It is the natural choice for very high QPS retrieval, for workloads where the embedding side already lives on GPU (so the vectors never leave HBM), or for latency critical RAG over mid scale corpora.

Your answer

**CAGRA** (CUDA Accelerated Graph Index) is NVIDIA's GPU-native ANN algorithm, shipping inside the **RAFT** library of GPU primitives. Conceptually it is a graph based ANN like HNSW: each vector is a node, edges connect near neighbors, search is a greedy walk. The difference is that the graph and the distance computations all live in GPU HBM (the GPU's on-chip memory), and the search exploits GPU parallelism heavily. **What it buys you.** Per-query throughput is roughly 5-10x higher than the best CPU HNSW at the same recall, on the same hardware budget. The win is largest when: - The vectors are high-dim (1024+), so per-distance computation cost dominates. - The QPS is high enough that GPU batching helps (CAGRA processes many queries in parallel). - The workload is read heavy with a stable index (CAGRA's build is less incremental friendly than HNSW). **The catches.** Two main ones. *GPU memory is small.* An H100 has 80GB of HBM. A billion 1024-dim float32 vectors is 4TB, so the index does not fit on one GPU; you either shard across many GPUs (operationally complex) or use compression (PQ on GPU). CPU boxes routinely have 256GB to 1TB of RAM at a fraction of the cost. *GPU instances are expensive per byte.* An A100/H100 cloud instance costs 5-10x what a comparable CPU instance costs per byte of RAM. So if the binding constraint on your bill is storage cost (which it usually is at billion-vector scale), GPU-ANN makes it worse, not better. **When to reach for it.** Milvus and Vespa both ship CAGRA integration in 2026. Pick it when your binding constraint is QPS or per query latency, not when it is storage cost. Natural fits: high QPS retrieval backends, workloads where embeddings are already generated on GPU and stay in HBM, latency critical RAG over mid scale corpora (1M-100M vectors).

**CAGRA and the RAFT library.** RAFT (Rapids Accelerated Framework Toolkit) is NVIDIA's library of GPU numerical primitives, of which CAGRA (CUDA Accelerated Graph Index) is the ANN component. CAGRA builds a small world graph similar in spirit to HNSW but with a flatter structure designed for the GPU's parallel execution model. Both the graph and the vectors live in GPU HBM during serving; the search kernel exploits the GPU's thousands of cores to evaluate many distance computations in parallel. **The throughput win.** At the same recall (typically measured at recall@10 = 0.95), CAGRA delivers 5-10x higher QPS than the best tuned CPU HNSW on the same hardware budget. The win compounds with vector dimensionality: at 1024 dims the gap is 5x, at 4096 dims it widens to 10-15x because the per-distance cost dominates and GPUs are dramatically better at fused multiply-adds. Per-query latency is also lower (often sub-millisecond at small QPS), but the bigger win is batched QPS where the GPU's parallelism shines. **The operational catches in detail.** 1. **GPU HBM is small and expensive.** An H100 has 80GB of HBM; an H200 has 141GB; an MI300X has 192GB. The largest single-GPU box is bounded by these numbers. A 1B-vector index at 1024 dims float32 is 4TB raw, which requires 25-50 GPUs to hold even with aggressive sharding. CPU boxes with 1TB of RAM are routine and dramatically cheaper per byte. 2. **Cost-per-vector inversion.** Cloud GPU instances cost 5-10x what CPU instances cost per byte of memory. For storage bound workloads, GPU-ANN is the worse economic choice. 3. **Less incremental friendly.** CAGRA's build is faster than CPU HNSW but the incremental-insert story is weaker. Workloads with continuous high rate writes still prefer CPU HNSW or hybrid approaches. 4. **Index portability.** A CAGRA built index is not directly compatible with CPU only deployments; switching off GPU means rebuilding. **The right workloads.** - **High-QPS retrieval.** Public-facing

TL;DR

CAGRA (NVIDIA's GPU graph-ANN, inside the RAFT library) gives you roughly 5-10x QPS over CPU HNSW at the same recall.

Memory aid

Easy to grasp

A CPU is like one really smart accountant doing additions one at a time. A GPU is like a stadium full of slightly dumber accountants who can all add at the same time. For vector search, where you need to compute thousands of small dot products, the stadium wins by a huge margin, often 10x faster. **But** the stadium has a tiny private vault, way smaller than the accountant's filing cabinet, and renting the stadium is much more expensive per square foot. So you bring the stadium in when you have a lot of queries pouring in and need them answered now; you don't bring it in just to store a billion items cheaply.

Key concepts

CAGRA

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

GPU accelerated ANN has been around for a decade (Faiss had GPU indexes in 2017), but the 2023-2024 NVIDIA RAFT / CAGRA release was the moment GPU-ANN became a serious production option rather than a research curiosity. The interview signal is whether the candidate understands what GPU-ANN actually buys, what it actually costs, and when to reach for it.

The headline: 5-10x QPS over CPU HNSW at the same recall, paired with 5-10x worse cost-per-byte-of-storage. That makes CAGRA the right tool for QPS-binding or latency-binding workloads and the wrong tool for storage-binding workloads. Calibrating to that distinction is the whole answer.

What CAGRA actually is

CAGRA (CUDA Accelerated Graph Index) is a GPU-native graph ANN algorithm, published by NVIDIA in 2023 and shipped inside their RAFT library of GPU primitives. The graph structure is similar in spirit to HNSW (small world graph, greedy walk) but flatter (single or shallow hierarchy) to match GPU SIMT execution.

During serving, both the graph topology and the raw vectors live in GPU HBM. Each query launches a CUDA kernel that maintains a candidate list in shared memory, evaluates many candidate distances in parallel using tensor cores or SIMD float lanes, expands neighbors, and iterates until convergence. The end to end query cost is dominated by HBM bandwidth and the cost of the distance kernel, both of which are dramatically better on GPU than CPU.

The public CAGRA paper claims 4-25x query speedup over CPU HNSW depending on dimensionality and recall target, with the high end on 1024+-dim vectors at recall 0.95. Independent benchmarks (BAAI, Anthropic, Milvus) confirm the 5-10x range as the typical production win.

Why HBM is the binding constraint

The killer 2026 use case: GPU-resident embedding pipelines

Decision rubric for 2026

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Property	CPU HNSW	CAGRA (GPU)	DiskANN
Primary storage	RAM	GPU HBM	SSD
Per-query latency	~1-3 ms	~0.2-1 ms	~5-15 ms
Sustained QPS	100-1000/core	10k-100k/GPU	100-500/box
$/byte of storage	1x baseline	5-10x worse	0.1-0.3x baseline
Max single-node index	~1TB RAM	80-192GB HBM	Multi-TB SSD
Best for	Default production	High QPS, low latency, mid-scale	Cost first billion scale

Real products, models, and research that use this idea.

Milvus 2.4+ ships CAGRA as a first class index type, recommended for high-QPS RAG and recommendation backends.
Vespa offers CAGRA integration via its tensor framework for hybrid scoring at scale.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does CAGRA pair with PQ to handle indexes that exceed HBM?

RAFT exposes PQ encoding on the GPU; CAGRA-PQ compresses vectors to ~32-64 bytes each, letting a single H100 hold ~1-2B vectors. Recall drops are similar to CPU IVF-PQ at the same compression.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming GPU-ANN is universally better than CPU HNSW because GPUs are faster. The cost per byte of memory inversion means GPU-ANN is the wrong choice for cost first storage bound workloads; it shines only when QPS or latency is the binding constraint.

60 second bullets to scan on the way to the call.

CAGRA sits inside NVIDIA RAFT; it is GPU-native graph ANN
5-10x QPS over CPU HNSW at the same recall on equivalent hardware

Primary sources. Browse if you want the original framing.