Flashcard: what is a vector database, and why does RAG need one?
A vector database is a data store built around fast approximate nearest neighbor search over high dimensional embeddings, so RAG can fetch the top-k similar chunks in milliseconds.
Imagine you have a library where every book is placed at a coordinate based on its meaning, so books about similar topics sit close together on a vast multi-dimensional shelf. Now someone walks in and asks a question. You convert their question into a coordinate too, and you need to find the books closest to that point. Walking past every shelf would take forever in a real library. So the library has a clever map that lets you jump straight to the right neighborhood without checking every aisle. A vector database is that library plus that map. It stores millions of vectors and ships with a search algorithm that finds the closest ones in milliseconds. Without it, RAG would crawl, because comparing one query against a million embeddings the slow way is a slow way.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Vector databases are simultaneously the most named and least understood component in a RAG stack. Candidates list Pinecone, Weaviate, and pgvector by reflex without being able to say what those systems actually do that a normal database cannot. The interview useful answer is sharper: a vector database is a system whose central design decision is an approximate nearest neighbor index over high dimensional vectors, with everything else (metadata filters, payload storage, multi-tenancy, freshness) layered around that index.
If you internalize that picture, the rest of the topic falls into place. You can talk about why a million vector corpus does not fit a SQL ORDER BY similarity query, why HNSW and IVF exist, why recall is a probability rather than a guarantee, and why two products you might both call 'vector databases' can behave very differently when you ask them for tenant filtered top-k.
This deep dive defines vector databases precisely, walks through the algorithms that make them fast, names the 2026 vendor landscape, and explains the operational concerns that experienced engineers spend most of their time on once the basic indexing is working.
The problem a vector database solves
Retrieval in RAG is a top-k nearest-neighbor query over embeddings. You have one query vector and a corpus of chunk vectors that can number in the millions or billions. You want the k chunks whose vectors are most similar to the query under cosine similarity or dot product. The naive way is a brute force scan: compute the similarity against every stored vector, sort, return the top k. That is correct but linear in corpus size.
The arithmetic breaks fast. A million 1024-dimensional vectors stored as float32 is 4 GB of data. A brute force scan touches every byte of it on every query. Even with SIMD and memory bandwidth on a modern server, the latency is hundreds of milliseconds. At ten million vectors it is seconds. No interactive product can pay that on every request.
The vector database exists to make this query sub-linear. It precomputes an index structure at insertion time, then uses that structure at query time to skip the vast majority of the corpus and zoom into the neighborhood where the answer lives. The result is typically tens of milliseconds at any scale that fits in memory, and a small constant factor more on disk backed indexes. Approximation is the price: ANN is approximate nearest neighbor, not exact. In practice, recall above 95 percent is routine, and a downstream reranker can recover the rest.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Pinecone serverless hosting tens of billions of vectors with metadata filtering and sub-100ms search latency for production RAG.
- Qdrant running in single binary mode with HNSW and on-disk vectors, common in self-hosted enterprise RAG deployments.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does HNSW achieve sub-linear search, and what parameters control the recall latency tradeoff?
Walk through the multi-layer navigable small world graph; explain M, efConstruction, and efSearch, and how each one moves recall versus query time.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a vector database is just a regular database with a vector column. The defining feature is the ANN index (HNSW, IVF, ScaNN) that makes top-k search sub-linear; without it you have storage, not retrieval.
60 second bullets to scan on the way to the call.
What problem a vector database solves that a normal SQL database cannot
Why ANN search is needed instead of brute force cosine similarity
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.