Design the fallback behavior when Langfuse is unreachable during prompt fetch
Your prod app fetches prompts from Langfuse on every request via get_prompt(). Langfuse has a 30 second outage. Design the fallback so user facing requests do not fail, while still preserving the prompt version audit trail when Langfuse comes back.
Cache prompts in process with a TTL, fall back to a baked-in default on cache miss, tag every trace with the version id that actually ran, and add a circuit breaker on the fetch path.
Think of how a restaurant handles its supplier going offline. The kitchen keeps a small stock of every ingredient in the back fridge (the cache), so a 30-minute supplier outage is invisible to diners. If even the back fridge is empty, the chef has a default recipe that uses pantry staples (the baked-in default) so service never stops. The receipt for each meal records which ingredient batch was actually used, including a tag like 'pantry-default' if it came from the fallback. When the supplier comes back, the audit trail explains exactly what was served during the outage, even though no diner ever saw a problem.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The principle that drives this design is simple and strict: the user-facing request must never block on the observability backend. Langfuse is a soft dependency, and a 30-second outage in a soft dependency should never become a 30-second outage in the product.
The right design has three layers that compose. An SDK-level cache with a TTL absorbs short outages invisibly. A baked-in default prompt provides the final backstop for cold-start during a long outage. An audit-trail discipline tags every trace with the prompt version that actually ran, so the post-incident review can reconstruct exactly what users saw. A circuit breaker on the fetch path prevents the recovery from being worse than the outage.
This deep dive walks through each layer, the failure modes each one covers, the trade-offs in TTL length, the sentinel-id pattern for the baked default, and the post-incident audit query you should be able to run within a minute of Langfuse recovering.
Mental model: the user path reads from the cache (or the baked default); the audit path writes the version id of whatever was actually used. The two paths are independent, so the audit survives even when fetches fail.
Layer 1: SDK cache with TTL
Every modern observability SDK that serves prompts exposes an in-process cache with a configurable TTL. Langfuse's get_prompt(name, cache_ttl_seconds=60) is the canonical example.
How the cache behaves
On first call for a (name, label) pair, the SDK fetches from Langfuse and stores the prompt object in an in-process map keyed on (name, label). Subsequent calls within the TTL window return the cached object instantly. After the TTL expires, the next call re-fetches.
How the cache absorbs an outage
If the TTL is 60 seconds and the outage is 30 seconds, any request that previously fetched the prompt in the last 60 seconds is served from cache without ever touching the network. In a healthy production system where the same prompts are fetched dozens of times per second, this means essentially every request is served from cache and the outage is invisible.
Sizing the TTL
Longer TTL hides outages better but slows the propagation of intentional prompt updates. Common production settings:
- 30 seconds: fast propagation, smaller outages absorbed.
- 60 to 120 seconds: typical sweet spot.
- 5 to 10 minutes: aggressive outage hiding, requires an explicit cache-bust API for fast rollouts.
Cache-bust escape hatch
Most teams pair the TTL with an admin endpoint that forces an immediate refresh. This lets you push an emergency prompt change in seconds rather than waiting for the TTL to expire, while keeping the long TTL for outage absorption.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Langfuse Python and JS SDKs both expose cache_ttl_seconds on get_prompt() as the primary defense against fetch failures.
- LangSmith Hub prompts support similar in-process caching plus a fetch_with_default pattern for the same reason.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you size the TTL?
Pick the longest acceptable propagation delay for intentional prompt updates. Common 30 to 120 seconds. Pair with an explicit cache-bust endpoint for emergency rollouts when 60 seconds is too slow.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blocking the user request on Langfuse availability. A 30-second outage in the observability backend becomes a 30-second outage on every customer-facing call, which is the inverse of what observability is supposed to do.
60 second bullets to scan on the way to the call.
Why the user-facing path must not block on observability availability
How an SDK-level cache with TTL absorbs short outages
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.