Your invoice extraction pilot works on a frontier VLM, but finance says the unit economics fail at the contracted volume of one million documents a month. Design the production system: cost structure, accuracy assurance, human review, and how you keep quality from drifting.
Free · 2 AI evals / day
At a million documents a month, the system has to do two things at once: keep accuracy high enough that finance trusts the books, and keep the cost low enough that the contract is profitable.
The single biggest lever is routing. Most invoices come from a repeat set of vendors with predictable layouts. A small, cheap multimodal model handles the bulk; only documents that fail validation rules or come back with low confidence are escalated to the expensive frontier model.
The second lever is structural cost reduction on every call: render pages at the resolution where line items are legible, not the maximum supported; use batch APIs where end-to-end latency does not matter; cache extractions by content hash so reruns are free; and lean on the fact that the same vendor templates repeat month after month.
Accuracy comes from layered validation, not from picking the smartest model. A strict JSON schema constrains the output shape. Deterministic rules like totals reconciling and dates parsing catch most arithmetic and formatting errors before anything else looks at the document.
The system has four pillars: cost structure, accuracy assurance, human review, and drift control.
**Cost structure** starts with model routing. A cheap default model, often a smaller open VLM or a provider's mini tier, handles the high-volume clean majority. Documents that fail deterministic validation, score below a confidence threshold, or come from never-before-seen vendor templates escalate to a frontier model like Claude Sonnet 4.6 or Gemini 3.1 Pro. The frontier model sees the tail, not the traffic. On top of routing, three orthogonal levers stack: tune rasterization resolution to the legibility floor rather than the model maximum; use batch APIs for non-urgent volume since they price at 30 to 50 percent of synchronous; and cache extractions by page-image content hash, with template-aware prompts that exploit the long tail of repeat vendors who send near-identical layouts every cycle.
**Accuracy assurance** is layered. Structured output against a strict JSON schema constrains the response shape. Deterministic business rules catch errors that no model self-confidence score can match: line items sum to subtotal, subtotal plus tax equals total, dates parse, currency is consistent, tax IDs match country patterns. Rule failures are the primary escalation signal.
**Human review** is a budgeted queue sized from finance's error tolerance. Confidence thresholds are calibrated so the daily queue is humanly processable; dollar value adds an override so a misread ten-thousand-dollar invoice always gets eyes regardless of confidence. Reviewer corrections are captured as labeled data for the next fine-tune round.
Quick reference
TL;DR
Route cheap model first, escalate on validation failure or low confidence, layer deterministic rules, size the review queue from error tolerance, and gate releases on a golden set plus production audits.
Imagine a busy mailroom sorting a million letters a month. Most letters are simple: the address is clear, the stamp is right, an intern handles them. A few letters are weird: handwriting smudged, foreign postage, unusual addressee. Those go to the senior sorter who is slow and expensive. The intern handles the river of normal mail, the senior sorter handles the exceptions, and a supervisor spot-checks both. The mailroom also keeps a binder of tricky old letters and tests every new sorter against it before letting them work alone. Run the same pattern with two model tiers, deterministic checks, a budgeted human queue, and a golden test set, and a million invoices a month becomes affordable.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Pilot or proof of concept volumes under a thousand documents a month The fixed cost of routing and golden-set CI exceeds the savings; a single frontier model with light validation is enough
Domains where every document is genuinely unique and there is no template reuse Caching and template-aware routing give no leverage; the design simplifies to one model with stronger review
Regulated extractions where every record must be human-reviewed regardless of model confidence The review queue is the system; the model is just an assistance layer, and routing logic does not save the budget
Quick scan · 2–4 min
Everything important, quickly.
8 minutes: the portfolio framing, the four pillars, the routing lever, and how senior teams treat the review queue as a training-data source rather than a tax.
Real-world examples
Real products, models, and research that use this idea.
1Ramp and Brex run AP automation at this scale by routing most invoices through smaller in-house extractors and reserving frontier models like Claude Sonnet 4.6 for unusual layouts
2Stripe Tax handles invoice and receipt extraction across hundreds of jurisdictions using a similar tiered approach, with deterministic rules around currency and VAT formats as the primary escalation signal
3Mindee and Klippa offer document AI APIs that explicitly expose a cheap default model and an upgrade tier, mirroring the routing pattern at the product level
4QuickBooks Online's AP feature uses Gemini 3.1 Pro on documents where the in-house OCR confidence is below threshold, fitting the escalate-on-tail pattern
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide when to fine-tune the cheap model versus when to keep relying on the frontier escalation path?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes if the contract requires a specific accuracy SLA per dollar value bucket?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Sending every document to a frontier model because the pilot did, then watching unit cost stay constant while volume grows linearly, instead of routing the cheap model first and reserving the expensive one for the tail.