Explain what NVIDIA CAGRA buys you over CPU HNSW and the operational catch that decides whether you should reach for it.

Attempt it

RAFT and CAGRA are NVIDIA's GPU primitives for ANN. CAGRA is a graph based index conceptually similar to HNSW but designed for GPU parallel traversal; RAFT is the wider numerical primitives library that CAGRA sits inside.

The win: roughly 5-10x QPS over CPU HNSW at the same recall, especially on high-dim vectors where per-distance dot product cost dominates. At billion-vector scale on a multi-GPU server, the gap can widen to 10-20x.

The operational catch is twofold. (1) GPU HBM is smaller and more expensive than CPU RAM, an 80GB H100 holds far fewer vectors than a 1TB CPU box, so very large indexes either don't fit or require multiple GPUs and inter-GPU sharding. (2) GPU instances cost 5-10x what CPU instances cost per byte of memory, so GPU-ANN makes sense when QPS or per query latency is the binding constraint, not when storage cost is.

Milvus and Vespa ship CAGRA integration in 2026. It is the natural choice for very high QPS retrieval, for workloads where the embedding side already lives on GPU (so the vectors never leave HBM), or for latency critical RAG over mid scale corpora.

Your answer

Quick reference

TL;DR

CAGRA (NVIDIA's GPU graph-ANN, inside the RAFT library) gives you roughly 5-10x QPS over CPU HNSW at the same recall.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

A CPU is like one really smart accountant doing additions one at a time. A GPU is like a stadium full of slightly dumber accountants who can all add at the same time. For vector search, where you need to compute thousands of small dot products, the stadium wins by a huge margin, often 10x faster. **But** the stadium has a tiny private vault, way smaller than the accountant's filing cabinet, and renting the stadium is much more expensive per square foot. So you bring the stadium in when you have a lot of queries pouring in and need them answered now; you don't bring it in just to store a billion items cheaply.

Key concepts

CAGRA

Quick scan · 2–4 min

Everything important, quickly.

4 min: name RAFT/CAGRA + the throughput win + HBM and cost catches + Milvus/Vespa integration + the decision rubric (QPS-binding vs cost-binding).

Side-by-side

Property	CPU HNSW	CAGRA (GPU)	DiskANN
Primary storage	RAM	GPU HBM	SSD
Per-query latency	~1-3 ms	~0.2-1 ms	~5-15 ms
Sustained QPS	100-1000/core	10k-100k/GPU	100-500/box
$/byte of storage	1x baseline	5-10x worse	0.1-0.3x baseline
Max single-node index	~1TB RAM	80-192GB HBM	Multi-TB SSD
Best for	Default production	High QPS, low latency, mid-scale	Cost first billion scale

Real-world examples

Real products, models, and research that use this idea.

Milvus 2.4+ ships CAGRA as a first class index type, recommended for high-QPS RAG and recommendation backends.
Vespa offers CAGRA integration via its tensor framework for hybrid scoring at scale.
NVIDIA cuVS is the standalone library teams use to wire RAFT/CAGRA into custom serving stacks.
Production vLLM + CAGRA deployments at frontier AI labs keep embedding and retrieval co-located on the same GPU, eliminating CPU-GPU transfer.
BAAI and Anthropic published 2024-2025 benchmarks showing CAGRA reaching 5-10x QPS over CPU HNSW on 10M-100M-vector benchmarks at the same recall.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does CAGRA pair with PQ to handle indexes that exceed HBM?

Sign in or go Pro to see the approach for this follow-up.

QWhy is the GPU embedding co-location story the killer use case in 2026?

Sign in or go Pro to see the approach for this follow-up.

QWhat is the rough crossover QPS where CAGRA beats CPU HNSW on $/query?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.