A new enterprise customer onboards with an empty index and zero eval data — stand up their RAG and prove it works

Short answer·Medium·4.0 · 0·~3 min·Relevant atAmazon Apple

Attempt it

A new enterprise customer signs on. You have their document corpus but nothing else: no labeled questions, no relevance judgments, no usage history. Describe how you would stand up a working RAG system for them and, critically, how you would know it works before you have any real eval data.

Free · 2 AI evals / day

With a corpus but no labels, the plan has three phases: stand it up on defaults, prove it works with a bootstrapped eval set, then improve from real usage. **Phase one is defaults.** Ingest the corpus — parse, clean, chunk, embed, and upsert with metadata like source and date. Don't try to tune anything from nothing. Pick components known to be safe across domains: a strong general embedding model, hybrid retrieval (dense plus BM25 keyword search) so you catch both meaning and exact terms, and a cross-encoder reranker on the candidate set. That gives a credible day-one baseline. Phase two is proving it without real questions. Use an LLM to generate question/answer pairs grounded in the customer's own documents, then have a human spot-check and prune them so the gold set is trustworthy. Run that set through retrieval metrics like recall@k and faithfulness checks to get a baseline you can defend. Phase three is improvement. Instrument the live system from the first session — thumbs up/down, edits, escalations — so real feedback accumulates and lets you fine-tune the retriever or reranker on the customer's actual queries.

The framing I'd give is: defaults first, synthetic then human eval to verify, online feedback to improve. You can't tune from zero labels, so you sequence around that constraint. First, stand it up on safe defaults. Ingest and index the corpus — parse, clean, chunk with overlap, embed, upsert with metadata. Then reach for components that are good across domains rather than tuned for this one. Use a strong general embedding model, hybrid retrieval so dense semantics and BM25 exact-term matching both contribute, and a cross-encoder reranker on the top candidates. This is a defensible baseline without a single customer label. Prove it before real data exists by manufacturing an eval set from the corpus itself. Have an LLM generate question/answer pairs grounded in their documents, then put a human in the loop to spot-check and prune. The human gate is non-negotiable, because a purely self-graded synthetic set can bless a broken pipeline. Now you can baseline both halves of the system. Retrieval quality covers recall@k, MRR, and context precision/recall. Generation quality covers faithfulness and answer relevance. That is the RAGAS-style split. Finally, improve from the moment users arrive. Instrument feedback from session one — thumbs up/down, answer edits, escalations to a human, dwell time. That live signal is the real eval set the synthetic one stood in for. It lets you fine-tune the retriever or reranker on the actual query distribution and catch failures the synthetic questions never imagined.

TL;DR

Ship strong defaults (general embeddings + hybrid retrieval + reranker), bootstrap an eval set by LLM-generating Q/A from their docs with human review, then improve from live feedback.

Memory aid

Easy to grasp

Imagine opening a library branch in a town you've never visited, with no idea what people will ask. You can't custom-tailor anything yet, so you start with a librarian who's good everywhere and a sensible filing system. To check it works before any visitors arrive, you read some of the books yourself and write practice questions whose answers you know are in there — then a colleague double-checks your practice questions are fair. You quiz the librarian on those and see how often they find the right shelf. Once real visitors start asking, you watch which answers got a thumbs up and which sent people away frustrated, and you adjust. So it goes: safe defaults first, a homemade but vetted practice quiz to prove it works, then real visitor reactions to make it better.

Key concepts

Retrieval-Augmented Generation LLM System Design Embeddings

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

2–4 min · Everything important, quickly.

4-5 min: the defaults then verify then improve arc, why you can't tune from zero, the synthetic eval with a human gate trick, the retrieval versus generation metric split, and the live-feedback loop.

Real products, models, and research that use this idea.

RAGAS-style synthetic test-set generation creates question/answer pairs from a customer's own documents to bootstrap evaluation.
Production RAG teams ship general embeddings plus hybrid BM25 retrieval and a Cohere-style reranker as the day-one default stack.
Support-bot deployments instrument thumbs up/down and escalation to a human from the first session to accumulate a real eval set.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you stop a synthetic eval set from being too easy or unrepresentative?

QOnce real feedback arrives, what do you fine-tune first — the embeddings or the reranker?

QHow would you design abstention so a cold-corpus bot doesn't answer confidently when retrieval is weak?

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Trusting an LLM-generated eval set without a human gate. A synthetic set scored by the same family of model that built it is self-graded — it can certify a broken pipeline as working.

60-second night-before-the-interview bullets. Scan these on the way to the call.

Primary sources. Skim if you want the original framing.