Design the product flow for a VLM that might misread a receipt total or a label

Start from one fact: the model is usually right but sometimes wrong. So the worst thing the product can do is take a reading and quietly act on it. The fix is to put the user in the loop. Show the value the model read inside an editable field, right next to the image, so a wrong number is easy to spot and fix. Then point at where the value came from — highlight the receipt line or crop it — so checking takes a glance, not a re-read. Next, use a confidence signal. When the model is sure, pre-fill the value and let the user move on. When it is unsure, flag the field and ask for a look, or send the image to a dedicated reader. For high-stakes flows like payments or medication doses, never auto-commit. Ask for explicit confirmation and log what happened, so a misread cannot become truth without someone seeing it.

The core principle is **graceful degradation**: every extracted value is a claim to be confirmed, never a fact to act on silently. The model is usually right, but "usually" is exactly the failure mode that hurts in a high-stakes flow. Design three layers. First, **surface and edit**: render each value in an editable field beside the image. Cite the region it came from — a crop or highlight of the receipt line — so the user verifies at a glance instead of re-reading the whole page. Second, confidence-gated behavior: above a threshold, pre-fill the value; below it, flag the field, prompt for confirmation, or route to a dedicated OCR pass or human reviewer. Third, a real fallback path for low confidence: re-capture, structured OCR, or escalation, not a silent best guess. For payments, dosages, and other irreversible actions, never auto-commit on the reading alone. Require explicit confirmation and keep an audit trail so a misread is always catchable. The shape of the answer is confirm, cite, and fall back — three levers that turn a possibly-hallucinated reading into a checkable claim.