Turn a convincing demo into numbers: evaluate multimodal RAG over figure heavy PDFs

Short answer·Hard·4.0 · 0·~3 min·Asked atFractal Analytics Rephrase Ai Zed

Attempt it

Your multimodal RAG system over figure heavy technical PDFs looks great in demos, but nobody can say how good it actually is. Design the evaluation harness: what you measure at each stage, how you score answers grounded in images, and how the harness gates releases.

Free · 2 AI evals / day

A demo looks great until you have to defend the system in front of a customer. The first job of an eval harness is to turn impression into numbers, and the second is to make those numbers diagnostic enough that a regression points at a specific component. Start by decomposing the system into stages. A multimodal RAG pipeline has at least three: retrieval (find the right pages and figures), generation (answer correctly given the retrieved context), and citation (the answer points back to the actual evidence). Each stage gets its own metric so a regression tells you where it lives. Build a labeled set of question-to-evidence pairs. For each question, write down which page or figure or table contains the answer. Stratify the set so it covers prose questions, chart questions, and table questions roughly in proportion to your real traffic mix. Aggregates lie when one slice is far harder than another. Measure retrieval with recall at k and MRR against the labeled evidence. Measure generation by giving the model the gold evidence and scoring faithfulness, with numeric hallucination tracked separately because charts invite confident wrong numbers. Measure citation accuracy by checking that the cited page actually contains the answer.

**The principle:** decompose before you score, because an end-to-end number cannot tell you which stage failed. The eval harness has four moving parts. **Labeled evaluation set.** A few hundred question-to-evidence pairs where evidence is a specific page, figure, or table. Deliberately stratified across prose questions (the answer lives in body text), chart questions (the answer requires reading a figure), and table questions (the answer requires reading a structured table). Visual retrieval usually lags text retrieval, and an aggregate metric hides that. **Stage-specific metrics.** - Retrieval: recall@k and MRR against the labeled evidence, reported per modality and per question type. This catches both miss-the-right-page and rank-the-right-page-too-low failures. - Generation given gold context: faithfulness (every claim, especially every number, is supported by the supplied pages), with numeric hallucination tracked as its own metric because charts invite confidently wrong numbers. Scored by a vision-capable LLM judge that sees the same images. - Citation accuracy: when the model cites page 7 for a claim, does page 7 actually contain that claim. Independent of whether retrieval was good or generation was right; a separate failure mode. **LLM judge calibration.** A vision-capable judge model (Claude Opus 4.7 or Gemini 3.1 Pro) sees the same images the system retrieved. Before the judge's scores are trusted, calibrate against 300 to 500 human-labeled examples per task to establish judge accuracy and detect biases. Re-audit the judge quarterly because model updates shift behaviour.

TL;DR

Decompose into retrieval, generation, and citation stages; stratify by question modality; calibrate a vision-capable LLM judge against human labels; gate CI on per-slice thresholds plus cost and latency.

Memory aid

Easy to grasp

Imagine a restaurant where customers complain the food is wrong sometimes. You cannot fix anything until you know whether the order was wrong (front of house), the cooking was wrong (kitchen), or the wrong dish made it to the table (runner). You taste-test each step separately. You write down what each menu item is supposed to look like and taste like, so you have a reference. You train a few judges to score consistently, then check them against the head chef occasionally. Only then do you say the food is N percent correct. A multimodal RAG eval harness works the same way. Decompose, label, calibrate, then score.

Key concepts

Multimodal AI Retrieval Augmented Generation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Multimodal RAG over figure-heavy PDFs is one of the most demoable AI applications in 2026. A user uploads a stack of technical documents, asks a question that depends on a chart on page 47, and the system returns the right answer with a citation. Demos go well, and then the engineering team is asked: how good is it, actually? The honest answer in most cases is nobody knows, because the team built a demo, not an eval.

This answer lays out the eval harness as a real piece of engineering. The structure follows what production teams converge on after a quarter or two of trying to defend their numbers to customers: decompose the system into stages, build a labeled set stratified by question modality, calibrate a vision-capable LLM judge against human labels, and gate releases in CI on per-slice thresholds plus cost and latency.

Why stage decomposition is the foundational design choice

A multimodal RAG pipeline has at least three components that can fail independently: a retriever that selects pages and figures from the corpus, a generator that produces an answer from the retrieved context, and a citation mechanism that points each claim back to specific evidence. An end-to-end accuracy number cannot tell you which of these failed when a regression appears.

Decomposition fixes this by attaching a metric to each stage. Retrieval is measured against labeled evidence: did the retriever return the page that actually contains the answer in its top-k. Generation is measured given gold context: if we hand the model the correct evidence, does it produce a correct, faithful answer. Citation is measured as a separate check: does the page the model claims as evidence actually contain the supporting content.

This decomposition pays off on every regression. The new chunking strategy regressed end-to-end accuracy; was it the retriever or the generator? With stage-level metrics, you see retrieval recall dropped for chart questions and generation faithfulness held steady. The fix lives in the chunking, specifically in how charts are extracted, and the team finds it in hours rather than days.

The alternative, one number, leaves the team guessing. Production teams that have lived through both will defend decomposition aggressively because they have paid the debugging tax of the alternative.

Labeled set: stratification is the design

Vision-capable LLM judges, calibrated against humans

CI gating and cost-quality balance

Production realities and 2026 model lineup

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic's evaluation cookbook for Claude Opus 4.7 explicitly recommends stage-level decomposition and human-calibrated LLM judges for multimodal RAG over technical documents
RAGAS and Patronus AI evaluation harnesses now ship visual variants for figure-grounded answers, reflecting industry adoption of the stage-decomposed approach

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you build the labeled set efficiently when expert labelers are scarce?

Bootstrap with a smaller seed set, generate candidate question-evidence pairs by prompting Claude Opus 4.7 against your corpus, and have experts review and correct rather than label from scratch. Active learning across early evaluation runs surfaces the highest-information examples to label next, stretching the budget.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Building one end-to-end accuracy number and using it as the release gate, then never being able to tell whether a regression was caused by the retriever, the generator, or the prompt.

60 second bullets to scan on the way to the call.

Why an end-to-end metric is insufficient for debugging RAG regressions
How to label question-to-evidence pairs across prose, chart, and table modalities

Primary sources. Browse if you want the original framing.