A new enterprise customer onboards with an empty index and zero eval data: stand up their RAG and prove it works

Short answer·Medium·4.0 · 0·~3 min·Asked atElastic Glean·Relevant atAmazon Apple

Attempt it

A new enterprise customer signs on. You have their document corpus but nothing else: no labeled questions, no relevance judgments, no usage history. Describe how you would stand up a working RAG system for them and, critically, how you would know it works before you have any real eval data.

Free · 2 AI evals / day

With a corpus but no labels, the plan has three phases: stand it up on defaults, prove it works with a bootstrapped eval set, then improve from real usage. **Phase one is defaults.** Ingest the corpus — parse, clean, chunk, embed, and upsert with metadata like source and date. Don't try to tune anything from nothing. Pick components known to be safe across domains: a strong general embedding model, hybrid retrieval (dense plus BM25 keyword search) so you catch both meaning and exact terms, and a cross-encoder reranker on the candidate set. That gives a credible day-one baseline. Phase two is proving it without real questions. Use an LLM to generate question/answer pairs grounded in the customer's own documents, then have a human spot-check and prune them so the gold set is trustworthy. Run that set through retrieval metrics like recall@k and faithfulness checks to get a baseline you can defend. Phase three is improvement. Instrument the live system from the first session — thumbs up/down, edits, escalations — so real feedback accumulates and lets you fine-tune the retriever or reranker on the customer's actual queries.

The framing I'd give is: defaults first, synthetic then human eval to verify, online feedback to improve. You can't tune from zero labels, so you sequence around that constraint. First, stand it up on safe defaults. Ingest and index the corpus — parse, clean, chunk with overlap, embed, upsert with metadata. Then reach for components that are good across domains rather than tuned for this one. Use a strong general embedding model, hybrid retrieval so dense semantics and BM25 exact-term matching both contribute, and a cross-encoder reranker on the top candidates. This is a defensible baseline without a single customer label. Prove it before real data exists by manufacturing an eval set from the corpus itself. Have an LLM generate question/answer pairs grounded in their documents, then put a human in the loop to spot-check and prune. The human gate is non-negotiable, because a purely self-graded synthetic set can bless a broken pipeline. Now you can baseline both halves of the system. Retrieval quality covers recall@k, MRR, and context precision/recall. Generation quality covers faithfulness and answer relevance. That is the RAGAS-style split. Finally, improve from the moment users arrive. Instrument feedback from session one — thumbs up/down, answer edits, escalations to a human, dwell time. That live signal is the real eval set the synthetic one stood in for. It lets you fine-tune the retriever or reranker on the actual query distribution and catch failures the synthetic questions never imagined.

TL;DR

Ship strong defaults (general embeddings + hybrid retrieval + reranker), bootstrap an eval set by LLM-generating Q/A from their docs with human review, then improve from live feedback.

Memory aid

Easy to grasp

Imagine opening a library branch in a town you've never visited, with no idea what people will ask. You can't custom-tailor anything yet, so you start with a librarian who's good everywhere and a sensible filing system. To check it works before any visitors arrive, you read some of the books yourself and write practice questions whose answers you know are in there — then a colleague double-checks your practice questions are fair. You quiz the librarian on those and see how often they find the right shelf. Once real visitors start asking, you watch which answers got a thumbs up and which sent people away frustrated, and you adjust. So it goes: safe defaults first, a homemade but vetted practice quiz to prove it works, then real visitor reactions to make it better.

Key concepts

Retrieval-Augmented Generation LLM System Design Embeddings

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Cold start is one of the most honest tests of whether someone has actually shipped RAG. There's no clever model to name and no dataset to point at — you have a pile of the customer's documents and a promise to make them useful. The interviewer wants to see how you reason when the usual crutch, labeled data, is gone.

The failure mode the question is probing is twofold: people either freeze ("I can't tune anything without labels") or over-build (elaborate fine-tuning on data that doesn't exist). The strong answer sequences around the constraint — robust defaults to ship, a manufactured but vetted eval to prove it, and a feedback loop to improve — and is honest about what each stage can and can't promise.

Why you ship defaults instead of tuning

With zero labels, every tuning decision is a guess dressed up as engineering. There's nothing to validate against, so optimizing embedding choice or chunk size on day one is overfitting to your assumptions about a corpus you barely know.

The right move is to pick components for robustness across domains, not fit to this one. A strong general-purpose embedding model is calibrated on broad data and won't fall over on an unfamiliar corpus. Hybrid retrieval — dense vectors fused with BM25 sparse search via reciprocal rank fusion — covers two failure modes at once: dense handles paraphrase and semantics, sparse handles the exact tokens (part numbers, error codes, acronyms) that embeddings blur. A cross-encoder reranker on the top candidates buys precision cheaply because it only scores a short list.

Ingestion comes first and is unglamorous but decisive. Parse and clean the documents, chunk with sensible overlap, attach metadata (source, section, date) for later filtering, embed, and upsert. Most cold-start quality problems trace back to bad parsing or chunking, not to model choice — so the boring step is where the leverage is. The output of this phase is a defensible baseline you can stand behind without a single customer label.

Manufacturing an eval set from the corpus itself

Baselining retrieval and generation separately

The feedback loop and the honest caveat

Setting honest expectations with the customer

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

RAGAS-style synthetic test-set generation creates question/answer pairs from a customer's own documents to bootstrap evaluation.
Production RAG teams ship general embeddings plus hybrid BM25 retrieval and a Cohere-style reranker as the day-one default stack.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you stop a synthetic eval set from being too easy or unrepresentative?

Seed hard cases deliberately: multi-hop questions, exact-term lookups, and unanswerable questions to test abstention, plus human pruning to remove questions the generator made trivially answerable.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Trusting an LLM-generated eval set without a human gate. A synthetic set scored by the same family of model that built it is self-graded — it can certify a broken pipeline as working.

60 second bullets to scan on the way to the call.

The cold-start sequence: defaults, then verify, then improve
Which retrieval defaults are robust across domains and why

Primary sources. Browse if you want the original framing.