Retry with exponential backoff, circuit breaker on repeated failures, and a structured tool-error message to the LLM so it can choose an alternate path instead of silently retrying forever.
Imagine you are working with a friend who sometimes does not answer the phone. The first time the line is busy, you try again in a few seconds. The second time, you wait a bit longer. If they have not answered after a few tries, you stop calling for a while so you do not flood their phone, and you decide whether to try someone else, do the task yourself with what you know, or ask for help. You do not pretend they answered when they did not, because then you would tell people the wrong thing. And you do not give up your whole day just because one friend is not picking up. An agent's tool-failure handling works the same way.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A third-party API tool at 99 percent uptime fails for several minutes every day on average, and longer during the occasional sustained outage. An agent that uses this tool in production has to keep working through those failures without lying about them. Tool fault tolerance is one of the engineering details that separates a prototype agent from a production agent.
The correct answer is a three-layer defence: retry with exponential backoff inside the tool wrapper, a circuit breaker that opens after repeated failures, and a structured tool-error message returned to the LLM when the tool is unavailable. Each layer handles a different timescale of failure, and each is necessary because each addresses a class of failure the others cannot.
The rest of this explanation walks the three layers in detail, examines the three wrong options to understand why each fails, covers the production discipline that makes the design work in real deployments, and closes on how the tool-error contract differs from traditional microservice fault tolerance.
Layer one: retry with exponential backoff
The first layer lives inside the tool wrapper and never bothers the agent. When a tool call fails for a transient reason (TCP reset, brief rate limit, network jitter, momentary timeout), the wrapper waits a short interval and tries again. The wait grows on each attempt, typically doubling: 500ms, 1s, 2s. Cap at three attempts total.
The exponential part matters during partial outages. If a dependency is already struggling, fixed-interval retries from many concurrent callers can compound the load and make the outage worse. Exponential backoff spreads retries out, giving the dependency room to recover. Adding jitter to the backoff (randomising each delay by 10 to 20 percent) prevents callers from synchronising on the same retry schedule.
The cap at three attempts matters because more retries do not help transient failures and do hurt the latency budget of permanent ones. After three attempts, the wrapper concludes this is not a transient blip and escalates to the next layer. A call that fails three times in three seconds is not going to succeed on the fourth.
Idempotency is the precondition for safe retry on mutating tools. A send-email tool that gets two requests with the same idempotency key must send exactly one email. Without idempotency, a retry of a call that actually succeeded but timed out before returning the response causes double execution. For read-only tools the issue is moot. For mutating tools, missing idempotency keys is one of the most common production bugs, and it shows up as duplicated effects right after a network blip.
Observability on this layer is simple: emit a metric per retry attempt, tagged by tool name. Dashboards show retry rates over time, which is the leading indicator of dependency degradation. A spike in retries at the wrapper layer is usually visible minutes before the dependency's own status page acknowledges an incident.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's tool-use documentation explicitly recommends returning structured error responses to the model so it can reason about tool failure rather than getting silently misled.
- Netflix's Hystrix and the modern resilience4j library encode the retry plus breaker plus fallback pattern that originated in microservices and translates directly to agent tool wrappers.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow should the tool-error message be shaped so the LLM actually uses it correctly?
Structured, named, and instructive. Include the tool name, a short human-readable reason, and a hint about what the model can do next ('try tool X for similar data' or 'proceed without this data but note the gap to the user'). Train the model on a few-shot example or fine-tune for the pattern. A generic 'error' string forces the model to improvise; a structured message gives it a clean path.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Silent retry forever or returning fake empty results. Both hide the failure from the agent and the user, producing stalled tasks or confidently wrong answers.
60 second bullets to scan on the way to the call.
Name the three layers: retry, circuit breaker, structured tool-error message.
Explain which timescale each layer addresses.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.