A third-party tool with 99 percent uptime is going to be unavailable sometimes. The agent has to keep working when that happens, without lying about it.
The right design has **three layers**, each handling a different timescale of failure.
**Layer one: retry with exponential backoff.** Inside the tool wrapper, the first call fails. Wait a short time, try again. If it fails, wait a bit longer, try again. Cap at three attempts. This handles transient blips like a TCP reset or a brief rate limit. The agent never sees these because they resolve inside the wrapper.
**Layer two: circuit breaker.** If the tool keeps failing across many calls, the wrapper marks it unavailable for a cooldown period (often 30 seconds). During the cooldown, calls fail immediately without trying the real endpoint. This stops the agent from hammering a dead dependency and gives the dependency time to recover.
**Layer three: tool-error message to the LLM.** When the tool is unavailable, the wrapper returns a structured error to the agent: 'this tool is currently down'. The LLM sees this and can choose: try a different tool that does something similar, fall back to its own knowledge with a caveat that the data may be stale, or escalate to a human. The agent never silently retries forever, and never pretends to have a result it does not.
The wrong options each fail. Infinite retry stalls the task. Silent empty results poison the reasoning. Taking the agent offline overreacts.
Three layers, each handling a different timescale.
**Retry with exponential backoff** inside the tool wrapper, capped at typically three attempts with delays like 500ms, 1s, 2s. This handles transient blips: TCP resets, brief rate limits, momentary network jitter. Resolved inside the wrapper, never bothers the agent.
**Circuit breaker** opens after repeated failures across multiple calls. The threshold is often 5 failures in 30 seconds, the cooldown is around 30 seconds. While open, the breaker fast-fails new calls without touching the real endpoint. This stops the agent from hammering a dead dependency and gives upstream time to recover. After cooldown, a half-open state probes the dependency with one real call before fully reopening.
**Structured tool-error message** to the LLM when the tool is unavailable: a clear, machine-readable error explaining the tool is currently down, with no fake data and no silent success. The LLM reads this and chooses an alternate path: a different tool that does something similar, its own knowledge with an explicit caveat that the live data is unavailable, or an escalation to a human via a HITL handoff.
The critical contract is that the tool-error message must be **honest**. Silent retry forever stalls the task at random. Returning fake empty results lets the agent assemble a confidently wrong final answer. Taking the whole agent offline overreacts to a single dependency outage.
Production adds observability: every retry, breaker open, and breaker reset emits a metric so dashboards show real-time tool health and alerts fire when failure rates cross thresholds.
Quick reference
TL;DR
Retry with exponential backoff, circuit breaker on repeated failures, and a structured tool-error message to the LLM so it can choose an alternate path instead of silently retrying forever.
Imagine you are working with a friend who sometimes does not answer the phone. The first time the line is busy, you try again in a few seconds. The second time, you wait a bit longer. If they have not answered after a few tries, you stop calling for a while so you do not flood their phone, and you decide whether to try someone else, do the task yourself with what you know, or ask for help. You do not pretend they answered when they did not, because then you would tell people the wrong thing. And you do not give up your whole day just because one friend is not picking up. An agent's tool-failure handling works the same way.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A tool that is part of a critical safety check where any data gap invalidates the entire output. Graceful degradation around a critical safety check produces unsafe partial answers; the right design is to terminate the task and escalate, not to continue with a gap.
A read-only local computation that cannot fail in any realistic sense, like a deterministic string transform. There is no failure to handle; retry and breaker add complexity without benefit.
A latency-critical voice agent where retries push the response past the user-tolerable budget. Retries add wall-clock time, and a single fail-fast with an immediate fallback path is better than the slow path of retry then fallback.
Quick scan · 2–4 min
Everything important, quickly.
Open with the framing that tool failure is a state the agent has to reason about, not an exception. Walk the three layers: retry with exponential backoff for transient blips, circuit breaker with open/half-open/closed states for sustained outages, structured tool-error message that the LLM can read and route around. Emphasise that the error must be honest. Cover idempotency keys on retries of mutating tools, observability on retries and breaker events, and the discipline of teaching the model how to respond to tool errors via the prompt or fine-tuning.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's tool-use documentation explicitly recommends returning structured error responses to the model so it can reason about tool failure rather than getting silently misled.
2Netflix's Hystrix and the modern resilience4j library encode the retry plus breaker plus fallback pattern that originated in microservices and translates directly to agent tool wrappers.
3LangGraph and LangChain ship retry decorators and circuit-breaker abstractions for tool wrappers, treating tool fault tolerance as a first-class concern in the framework.
4Production browser-using agents like Anthropic's computer use and OpenAI's Operator handle browser-tool failures with retries plus a structured error to the model, which can then re-plan around a missing capability.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow should the tool-error message be shaped so the LLM actually uses it correctly?
Sign in or go Pro to see the approach for this follow-up.
QThe third-party tool is the only one that does what you need. There is no alternative. How should the agent degrade?
Sign in or go Pro to see the approach for this follow-up.
QHow do you decide the retry count, backoff schedule, and breaker thresholds without making them arbitrary?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Silent retry forever or returning fake empty results. Both hide the failure from the agent and the user, producing stalled tasks or confidently wrong answers.