A downstream API changes its schema while your agent is live in production. How do you handle it?
Define schema drift in an agentic system. Describe the failure mode it produces and the three main defensive strategies for handling it in production.
Schema drift is when a downstream API's contract changes under a running agent, breaking tool calls; you survive it with versioned schemas, runtime validation, and graceful fallbacks.
Imagine you gave a friend a form to fill out whenever they order your favourite coffee. The form asks for your name and your drink. One day the cafe quietly redesigns the form, renaming a box and adding a new required one, but nobody tells your friend. Your friend keeps filling out the old form from memory. Sometimes the cafe rejects it outright, which is annoying but at least obvious. Worse, sometimes it accepts the old form and just makes the wrong drink, and you only find out after the first sip. An AI agent talks to other software through forms like this, called tool schemas. When the other side changes its form without warning, the agent keeps using the old one. The fix is to check the form is still current, keep a known-good version, and have a backup plan when something looks off.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Define schema drift as a diverging contract, separate the explicit and silent failure modes, then walk the four-part defence: version the schema, validate responses at runtime, catch drift in CI from a single source of truth, and degrade gracefully. Close with detection via validation-failure alerting.
from pydantic import BaseModel, ValidationError
class WeatherResponse(BaseModel): # single source of truth, versioned
temp_c: float
condition: str
def call_tool(raw):
try:
return WeatherResponse.model_validate(raw) # strict, not just json.loads
except ValidationError:
metrics.incr("tool.schema_validation_fail", tags={"tool": "weather"})
return fallback_weather(raw) # degrade, do not crash the taskReal products, models, and research that use this idea.
- An agent built on the Anthropic SDK or OpenAI tool use pins each tool to a versioned API and regenerates its JSON Schema from the provider's OpenAPI spec in CI, so a renamed field fails the build instead of production.
- Pact-style consumer-driven contract tests let an agent team publish the response shape it depends on, so the upstream service's pipeline fails before a breaking change ships.
- LangGraph and the Model Context Protocol expose tool schemas the agent can fetch at startup, enabling a runtime refresh that detects drift before the first real call.
- Observability stacks like Langfuse, Arize, and LangSmith track per-tool validation-failure rates, so an alert fires when a downstream change spikes errors on one tool while the rest stay healthy.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you detect silent schema drift, where a renamed or dropped field still parses but carries wrong data?
QWhere should the canonical tool schema live so the agent definition never diverges from the real API?
QWhat is the tradeoff between pinning to a fixed API version and fetching the live schema at runtime?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating schema drift like an ordinary code bug. The defect lives in a third party's API, so there is no agent-side code change to point at, and silent versions produce no exception at all.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.