When is using an agent architecture an engineering mistake, even when LLMs are involved?
Describe the conditions under which choosing an agent architecture is an engineering over complication, even for an LLM powered task. What alternative is appropriate and why?
An agent is the wrong tool when the execution path is fixed at design time. A chain is cheaper, faster, and predictable when nothing needs dynamic routing.
Imagine you need to make tea the same way every morning: boil water, add a bag, pour, steep, remove. You do not hire a chef who decides each step fresh, asks what to do next, and might wander off to grind beans. You just follow the recipe. An agent is the chef: smart, flexible, and great when the next move genuinely depends on what you find. But for a fixed recipe, that flexibility is wasted. Worse, the chef pauses to think before every action, so the tea takes longer and costs more, and some mornings the chef gets confused and makes ten cups. A chain is the recipe card. When you already know the steps and their order, the recipe wins on speed, cost, and the simple comfort that tomorrow's tea looks exactly like today's.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The misconception is seductive: the task uses a large language model, therefore build an agent. The reasoning feels modern, and agents are the headline architecture of the moment. But agency is not a property of using an LLM. It is a property of the workload. An agent is the right architecture only when the next action genuinely cannot be known until the previous result comes back.
This distinction matters because the agent loop is the single most expensive control structure you can choose. It adds a reasoning call on every turn, it lets the path branch in ways you did not enumerate, and it makes the total cost and latency of a run a runtime variable rather than a design-time constant. Those are real costs, and they are only worth paying when the workload actually demands dynamic decisions.
When the execution path is fixed at design time, an agent is not just unnecessary. It is actively worse on the engineering axes that decide whether a system survives production: predictability, cost, latency, and debuggability. The correct tool for a fixed path is a chain, also called a pipeline or a fixed directed graph. This explanation walks through what an agent really costs, the precise condition that makes it over-engineering, what to build instead, and the one question that settles the choice in a design review.
What an agent actually costs
An agent is an LLM in a loop. On each turn it reads the accumulated state, reasons about the next move, calls a tool, captures the observation, and decides whether to continue. That extra reasoning call is the price of flexibility, and you pay it every single turn. A chain pays no such tax: its control flow is fixed, so the model is only invoked for the steps that genuinely need generation.
Three costs compound with the number of turns. Latency compounds because each turn adds a full model round trip, so a ten-turn run is ten serialised calls stacked end to end. Token cost compounds super-linearly because the state grows with each observation and the whole transcript is re-read on every turn, so the eighth turn pays for the seven that came before it. Reliability degrades because a loop can misinterpret an error, repeat a broken call, and burn the entire step budget for zero progress.
There is a fourth, subtler cost: operational surface area. An agent needs a termination policy, a step budget, a cost cap, a repeated-action detector, and per-step tracing before it is safe to ship. Every one of those is code you write, test, and maintain. A chain needs none of them, because a straight-line graph cannot loop forever and cannot wander off its route.
None of these costs are wrong in themselves. They are the fair price of dynamic decision making, and on a task that truly branches they buy real capability. The mistake is paying that price when the workload gives you nothing in return, because the route was already known before the run began.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Document ingestion pipelines in LangChain LCEL run extract, chunk, embed, and store as a fixed graph with no loop, because the order never changes.
- Basic retrieval augmented generation over a knowledge base is a two-step chain of retrieve then generate, which production stacks like LlamaIndex express without any agent.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhere exactly is the line between a conditional chain and a true agent?
Frame it as branching factor and depth. A bounded, enumerable set of routes is a conditional graph. An open-ended action space with an unknown number of turns and runtime replanning is an agent. Argue the cost of generality only pays off past that boundary.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reaching for an agent because the task uses an LLM. Agency is justified by a dynamic execution path, not by the mere presence of a model in the system.
60 second bullets to scan on the way to the call.
State the precise condition under which an agent is over-engineering.
Name the chain or fixed pipeline as the correct alternative.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.