Build a screenshot driven UI testing agent that does not flake

Screenshot agents are flaky for four predictable reasons and each has a clean fix. **Where the click lands** depends on whether the agent's pixel coordinates line up with the live viewport. Device pixel ratio scaling on retina displays, or a screenshot taken at a different size from the browser, shifts every click. Capture and act in the same coordinate space, then take a follow-up screenshot to verify the action did what you expected. **When you capture** matters enormously. Fixed `sleep(2)` calls produce screenshots of spinners and half-rendered DOM. Wait on deterministic signals: network idle, an element becoming visible, a known text appearing. **What the model can read** depends on perception fidelity. Downscaling a 1080p screenshot to 512 by 512 makes small labels unreadable. Keep screenshots near native resolution and crop to regions when budget demands. **What you assert on** decides whether you flake on cosmetic changes. Pixel-equality breaks on every theme update. Assert on text content and element presence using the DOM or accessibility tree; reach for vision only where the DOM cannot see, like canvas or charts.

**Four flakiness sources**, each with a clear design choice that tames it. **Grounding.** The model emits click coordinates from a screenshot, and any mismatch between screenshot resolution, browser viewport, and device pixel ratio shifts every coordinate. Decide on one canonical coordinate space, screenshot in it, and act in it. After every click, capture a follow-up screenshot and verify the expected element changed; if not, retry with adjusted grounding. **Timing.** Fixed sleeps cause flake; deterministic waits do not. Wait for `networkidle`, for an element to be visible, or for a known text to appear in the DOM before capturing. Capture during animation and you screenshot a spinner. **Perception fidelity.** Down-scaling kills small UI: button labels, form field hints, error messages. Keep UI captures at or near native resolution; if token budget bites, crop to the relevant region instead of shrinking the whole frame. High detail mode is the right default for UI tasks. **Assertions.** Pixel-diff breaks on every font or theme change. Assert on semantic state: text content, element presence, ARIA role. Use the DOM or accessibility tree wherever the DOM can see; use the VLM only where the DOM cannot, such as canvas-rendered charts. Wrap with per-step retries, history eviction so old screenshots do not balloon token cost, and a replayable trace of screenshot, action, and outcome at every step.