Click any words you think contain an error. Click again to unmark.
Three errors: native 4K wastes tokens past the provider ceiling, append-only screenshot history overflows context within tens of steps, and prompt caching is neither automatic nor free.
Think of an agent's context window like a tiny office desk. Every screenshot is a thick photo album you drop on the desk after each step. Taking that photo in 4K does not help, because the office only reads pictures up to a certain size and shrinks the rest. Stacking every album you ever shot quickly buries the desk, even though you planned to take 200 photos. And the claim that the office files duplicate albums for free is wishful thinking: filing has rules, and any change to the older stack tears up the discount. Working agents take normal sized photos, keep only the most recent ones on the desk, and write a short note describing the older ones.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Computer-use agents look like a clean abstraction: the loop is screenshot, think, act, repeat. The flaws in this plan are the three places that abstraction lies. Pixels are not free, history does not compress itself, and caching does not refund tokens.
This deep dive walks through the token economics, the memory architecture that real agents use, and the prompt-caching mechanics that turn the cost model from a wild guess into something an FP&A team will sign off on.
Section 1: the cost of one screenshot
Claude's image-token estimate is straightforward: tokens approx width_px times height_px / 750. GPT-5.5 uses a tile-based model where a roughly 1024 px image is one base tile plus a small number of detail tiles. Gemini 3.1 Pro bills by tile count similarly. The exact formulas matter less than the order of magnitude.
A 1568x1080 screenshot under Claude:
A 3840x2160 screenshot at native 4K is over 11,000 tokens before downscaling, except the provider does downscale before billing, so you actually pay for the post-downscale size and waste only the upload and pre-processing. Either way, 1.5k to 2k tokens per screenshot is the real number to plan against.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's computer-use reference implementation for Claude Opus 4.7 and Sonnet 4.6 caps active screenshots at a small sliding window and trims older images from the message history
- OpenAI's GPT-5.5 computer-use product documents prompt caching at a 50 percent discount on stable prefixes, not free reuse
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide between a sliding window of recent screenshots and a structured text observation log?
Talk about which decisions need pixel-level evidence (clicking a specific button, reading a chart) versus which only need a state summary (already logged in, on page 3). The hybrid is keep the last K frames as pixels, summarize the rest, and only re-attach an older frame if the model explicitly requests it via a tool call.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating screenshots as free pixels rather than as the costliest tokens in the loop, then assuming caching reverses that cost without any prefix discipline.
60 second bullets to scan on the way to the call.
Token cost formula for a screenshot under Claude and rough cost under GPT and Gemini
What the provider's effective image input ceiling is and what happens above it
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.