Why the supervisor agent is often the dominant cost line in a supervisor topology

Flashcard·Medium·4.0 · 0·~30s·Asked atCognizant Neptune Ai Snowflake

Attempt it

The supervisor runs once per hop. Each call reads the full message log plus the agent roster plus its own routing prompt. As the workflow progresses the message log grows (every worker output adds to it), so each successive supervisor call costs more than the previous one. By turn 8 the supervisor is reading 8 turns of conversation just to pick the next agent.

Four fixes, roughly in order of effort. One, move the supervisor to a cheaper model (mini-tier or distilled router) since routing is closer to classification than to reasoning. Two, summarise the message log between supervisor calls so it does not grow unboundedly. Three, hand off with structured payloads instead of raw chat history so the supervisor's input stays bounded. Four, replace the LLM supervisor with a heuristic or a tiny fine-tuned router for the common cases and reserve the full LLM call for the long tail.

Your answer

The supervisor cost-dominance pattern surprises many teams because the supervisor 'does not do the real work'. The math says otherwise. **Why the supervisor compounds.** The supervisor runs once per hop. Each call reads three things: the supervisor's own routing system prompt, the agent roster description, and the message log so far. The first two are constant per hop. The third grows: every worker output adds to the log, so every subsequent supervisor call reads a longer log than the previous one. If the workflow runs 10 hops with workers producing 500 tokens each, by hop 10 the supervisor's input includes 9 prior worker outputs (about 4,500 tokens of message log) plus the system prompt plus the roster. The supervisor is reading 5,000+ tokens just to pick the next agent. The single-call cost looks reasonable; the cumulative cost across 10 hops is large. Meanwhile each worker runs maybe once or twice with a more bounded input. Per-call worker cost stays roughly stable; per-call supervisor cost grows linearly with hop count. **Four fixes in order of effort:** 1. **Move the supervisor to a cheaper model.** Routing is classification-shaped; a mini-tier model handles it well at 5-10 percent of the frontier cost. Easiest fix, biggest immediate win. 2. **Summarise the message log between supervisor calls.** Run a small summariser model on the log every few hops so the supervisor reads a condensed history instead of the raw transcript. 3. **Hand off with structured payloads instead of raw chat history.** The supervisor sees the latest worker summary, not the full output. Input stays bounded. 4. **Replace the LLM supervisor with a heuristic for common routing cases.** Rule-based or fine-tuned router for the modal cases, reserve the LLM call for the long tail.

**Supervisor cost dominance** comes from two compounding factors: the supervisor runs every hop, and its per-hop input grows linearly with conversation length. Worker cost stays bounded; supervisor cost compounds. **The mechanics.** Each supervisor call reads routing prompt (~500-1000 tokens, constant), agent roster (~200-500, constant), and message log (grows ~500-2000 per hop). By hop 10 the per-call input is ~10,000-20,000 tokens. Aggregated across 10 hops, supervisor cost dominates worker cost which stays bounded by task. **Four fixes, in order of effort and impact.** 1. **Model tiering.** Move the supervisor to a mini-tier model (gpt-4.1-mini, Haiku 4, Gemini Flash). Routing is classification-shaped; a smaller model handles it at 5-10 percent of frontier cost. Single highest-leverage fix; one config change. Cuts supervisor cost 80-90 percent for most crews. 2. **Message-log summarisation.** Insert a small summariser every N hops. The supervisor reads condensed history. The summariser cost is small relative to savings. 3. **Structured-payload handoffs.** Replace raw chat history forwarding with structured payloads (task result, key findings, status). Input stays bounded across hops. Bonus: also prevents role bleed. 4. **Heuristic routing for common cases.** Replace the LLM with a rule-based or tiny fine-tuned router for the modal cases. Reserve the LLM supervisor for ambiguous long tail. For many crews, 70-80 percent of routes are mechanical. Production systems layer all four. A 10-hop workflow that naively costs $1.50 in supervisor calls drops to $0.05 with full optimisation.

TL;DR

The supervisor runs every hop and reads a growing message log; its per-call cost compounds. Fix by tiering down the model, bounding input via summaries and structured payloads, or routing rule-based for common cases.

Memory aid

Easy to grasp

Imagine a manager who walks into every meeting having read every email from the entire project so far. On day one that is one email. On day twenty that is five hundred emails per meeting. Even though their job (decide who picks up the next task) is short, the reading time piles up enormously. Multi-agent supervisors work the same way: they run every turn and read all prior turns to decide. If you do not give them a shorter inbox (summaries, structured updates) or a faster reader (smaller model), their cost grows quietly until it dominates the bill.

Key concepts

Multi-agent systems Context engineering Inference optimization

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Multi-agent cost dynamics surprise teams who carry single-agent intuition into hierarchical workflows. Single-agent cost is roughly per-call input times call count, with both factors bounded by task design. Multi-agent supervisor cost compounds: the supervisor runs every hop, and each call reads a growing message log. Both factors scale with workflow length, and the product dominates the worker cost on anything past a few hops.

The cost dominance is invisible if you look only at per-call cost, the supervisor's calls look reasonable in isolation. It is glaring if you look at aggregate spend with a per-agent breakdown. Production teams that ship multi-agent systems without per-agent observability discover this pattern only when the monthly bill arrives.

This section walks through the compounding mechanism, the four-layer optimisation strategy (model tiering, input bounding via summarisation, structured-payload handoffs, hybrid rule-plus-LLM routing), the prompt-caching interaction, and the observability discipline required to catch this pattern before it ships.

The compounding cost mechanism

Every supervisor call reads three components.

Routing system prompt. The instruction set for the supervisor: how to decide which worker handles the next task, what format the routing decision takes, what edge cases to escalate. Typically 500-1000 tokens. Constant per hop.

Agent roster description. A list of available workers with role names, capabilities, and any per-worker constraints. Typically 200-500 tokens. Constant per hop.

Message log so far. Every prior worker output, often the user's original input, occasionally intermediate planner reasoning. This grows by 500-2000 tokens per hop as workers add outputs to the shared log.

The first two are bounded. The third compounds. By hop 10 in a moderate-complexity workflow, the supervisor's per-call input is 10,000-20,000 tokens. Per-call cost at frontier-tier pricing (around $15 per million input tokens for Claude Opus 4.5 or GPT-5) is around $0.15-0.30. Per workflow with 10 supervisor calls: $1.50-3.00 just for the supervisor.

Meanwhile each worker call has a bounded input (the worker's specific task plus relevant context, typically 1000-3000 tokens). Per-call worker cost at frontier-tier: $0.015-0.045. Each worker is invoked 1-2 times per workflow. Total worker cost across 4 workers: roughly $0.10-0.30.

The supervisor outspends all the workers combined by a factor of 5-15x. This is the inverse of what teams intuitively expect, the workers do the real work, so they should dominate the bill. The structural shape of the supervisor's compounding input flips the ratio.

Worker output spend. Routing decisions are short outputs (50-200 tokens), so output spend on the supervisor is small. Worker outputs are longer (500-2000 tokens) but worker invocation count is bounded. Output spend matters less than input spend on modern pricing where input is 5-10x cheaper per token than output but input volumes are 10x larger; the input side dominates.

Layer 1: model tiering

Layer 2 and Layer 3: bounding input growth

Layer 4: hybrid routing and the prompt-caching interaction

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

LangChain published in 2024 that their internal multi-agent benchmarks saw supervisor cost dominate workflow cost on runs past 8 hops, motivating per-node model tiering recommendations.
CrewAI hierarchical crews in 2026 production deployments commonly run manager_llm on Haiku or gpt-4.1-mini with workers on Sonnet 4.5 or gpt-5.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you set up a per-agent cost breakdown in production observability?

Tag every LLM call with an agent.name attribute (the speaker or node identifier). Configure the LLM observability backend (Langfuse, Datadog LLM Obs, Arize Phoenix) to group token-cost metrics by agent.name. Alert when any single agent exceeds a threshold percentage of total workflow cost. This surfaces supervisor dominance immediately.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Looking at the supervisor's per-call cost and concluding it is cheap, missing that the per-call cost grows linearly with conversation length and the call count grows with workflow complexity. Both compound.

60 second bullets to scan on the way to the call.

Why supervisor cost compounds while worker cost stays bounded
The three components of every supervisor call (routing prompt, agent roster, message log)

Primary sources. Browse if you want the original framing.