How prompt caching pays off disproportionately well in multi-agent systems

Flashcard·Medium·4.0 · 0·~30s·Asked atBraintrust Notion Vernacular Ai

Attempt it

Single-agent runs typically vary the input every call (different user message, different tool result). Cache hits happen on system prompts and stable tool definitions but the dynamic suffix is large.

Multi-agent runs have a different shape. The supervisor's prompt and the agent roster description are identical every hop; only the latest message changes. That is exactly the cached prefix plus dynamic suffix pattern caching is designed for. Worker agents reused across hops also hit cached prefixes for their system prompts and tool definitions. A chatty 8-hop supervisor run can see the supervisor's cached read price drop to 10 percent of the uncached price, which alone takes a meaningful slice off the bill. The discipline is: put the largest stable content (system prompt, agent roster, long tool schemas) at the front of every prompt and never shuffle it; let the per-turn delta sit at the suffix where it cannot break the cache.

Your answer

**Why single-agent gets less out of caching.** A single-agent call typically has a stable system prompt and tool definitions at the front and a changing user message at the back. Cache hits cover the system prompt; the dynamic part is what the model has to re-read every time. The savings are real but modest. **Why multi-agent supervisors get more.** A supervisor sends the same prompt every hop: - System prompt (identical). - Agent roster description (identical). - Long tool schemas (identical). - Conversation history (grows slowly, with mostly stable past content). - Latest worker output (the only large changing part). That is exactly the cached prefix plus dynamic suffix pattern caching is designed for. The stable prefix can be huge (5K to 10K tokens of agent-roster + tool-schemas) and is identical every hop. **Concrete impact.** Anthropic and OpenAI both price cached reads at about 10 percent of uncached reads. A chatty 8-hop run with a 5K-token stable prefix saves about 80 percent of the supervisor's input cost. On top of the downsize-supervisor lever, that compounds to 15 to 30 times less cost. **The discipline.** Put the largest stable content at the front. Never shuffle it. Let the dynamic per-turn delta sit at the back where it cannot break the cache.

**Single-agent caching is good; multi-agent caching is great** for a structural reason: the supervisor's prompt has a much higher ratio of stable prefix to dynamic suffix than a typical single-agent call. A single-agent call typically has a system prompt (stable, maybe 500 to 2K tokens) and a user message (dynamic, varies in length). Cache covers the system prompt; the dynamic part is the bulk of what changes call to call. Savings are modest because the cacheable share is small. A supervisor and workers run has a different shape on every hop: - System prompt: stable. - Agent roster description (one-line description per worker): stable. - Long tool schemas (one per available tool): stable. - Conversation history so far: mostly stable; only the latest worker output is new. That stable prefix can be 5K to 10K tokens. The dynamic suffix is the latest worker output, maybe 1K tokens. The cacheable share is 80 to 90 percent, not 30 to 50 percent. With cached reads priced at roughly 10 percent of uncached reads, the supervisor's per-hop input cost drops by 70 to 80 percent. **Worker reuse compounds the win.** Workers reused across hops also have stable system prompts and tool definitions. Each worker invocation hits cached prefixes for its own static content, adding another layer of savings. **The discipline that makes it work.** Put the largest stable content at the front of every prompt and NEVER shuffle it. Per-turn deltas go at the suffix. One shuffled token in the middle invalidates the entire downstream cache. Treat prompt order as load-bearing infrastructure, not a stylistic choice. Combined with the supervisor-downsize lever, prompt caching takes multi-agent cost from 'expensive enough to question' to 'a fraction of what it used to be.'

TL;DR

Multi-agent supervisors send a large stable prefix (system prompt plus agent roster) on every hop with only a small dynamic suffix; that is exactly the shape prompt caching rewards, and worker reuse compounds the win.

Memory aid

Easy to grasp

Imagine a courier service where every package gets the same 100-page company manual stapled to the front and a one-page customer note at the back. The first time the receiving warehouse processes a package, they read the whole thing. Every package after, they recognize the manual, skip to the back, and only read the note. That is prompt caching. In single-agent runs, the manual changes too often to skip. In multi-agent supervisor runs, the manual (system prompt plus agent roster) is identical every hop, so the warehouse skips to the note every time and the bill drops a lot. The trick is keeping the manual at the front and never editing it.

Key concepts

Multi-Agent Systems

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Prompt caching is one of the more underappreciated cost levers in multi-agent systems. The mechanism is simple (cached reads cost roughly 10 percent of uncached on Anthropic and OpenAI) but the structural reason it pays off disproportionately well for multi-agent is worth understanding: the supervisor and workers pattern produces prompts with an unusually high ratio of stable prefix to dynamic suffix, which is exactly the shape caching rewards.

This walkthrough explains the ratio, names the discipline that keeps caching working, lists the invalidation traps that silently undo the savings, and shows how worker reuse compounds the win. The goal is to get the supervisor's cache hit rate to 80 percent plus in steady state and keep it there.

Mental model: prompt caching is amortization. The first call pays for the cache; every subsequent call within the TTL pays a tenth. Multi-agent runs reuse the cache many times per task; single-agent runs typically do not.

The structural reason multi-agent benefits more

Single-agent prompt shape

A typical single-agent call:

System prompt: 1-2K tokens, stable.
Tool definitions: 0.5-2K tokens, stable.
User message: 0.5-3K tokens, dynamic.
Conversation history (if any): variable, partially stable.

Cacheable share: 30 to 50 percent of input tokens. Savings are real but bounded.

Supervisor hop prompt shape

A typical supervisor hop in a multi-agent run:

System prompt: 1-2K tokens, stable.
Agent roster description (one-line per worker plus capability summary): 1-3K tokens, stable.
Tool schemas (every tool available to the supervisor): 2-5K tokens, stable.
Conversation history of prior hops: 2-5K tokens, mostly stable; only the latest worker output is new each hop.
Latest worker output: 0.5-1.5K tokens, dynamic.

Cacheable share: 80 to 90 percent of input tokens. The savings compound across every hop.

Why the ratio matters

Cached reads price at roughly 10 percent of uncached. A prefix that is 30 percent of input gives you 0.7 * 1.0 + 0.3 * 0.1 = 0.73, a 27 percent savings. A prefix that is 85 percent of input gives you 0.15 * 1.0 + 0.85 * 0.1 = 0.235, a 76.5 percent savings. The ratio is doing the heavy lifting.

Why this is the natural shape, not a special case

The supervisor's job is to read the latest message and route. The roster, tools, and system prompt are part of who it is, not part of what changed this turn. Designed well, the supervisor's prompt has the largest part of its bytes in the stable prefix by default. The shape rewards caching without engineering effort to make it so.

The discipline that keeps the cache working

Worker reuse: the second compounding layer

Cache invalidation traps and production telemetry

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic's prompt caching docs use the supervisor and workers shape as the motivating example for cached prefixes.
OpenAI's Responses API and Agents SDK auto-cache prefixes above 1024 tokens, which fits the supervisor shape naturally.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do Anthropic's cache_control breakpoints differ from OpenAI's auto-caching?

Anthropic requires explicit cache_control: {type: 'ephemeral'} markers on the content block(s) that mark the cache boundary; you control where the breakpoint sits and can have up to 4 breakpoints per request. OpenAI auto-caches any prefix above 1024 tokens with no explicit markers; the API decides the breakpoint. Anthropic gives more control; OpenAI is more automatic. Both reward stable-prefix discipline.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Reordering the prompt or interpolating dynamic content into the middle of a stable prefix, which invalidates the cache on every call and silently doubles the per-hop cost.

60 second bullets to scan on the way to the call.

Why supervisor prompts have a higher cacheable share than single-agent prompts
Cached-read pricing (roughly 10 percent of uncached) on Anthropic and OpenAI

Primary sources. Browse if you want the original framing.