Design the fallback behavior when Langfuse is unreachable during prompt fetch
Your prod app fetches prompts from Langfuse on every request via get_prompt(). Langfuse has a 30 second outage. Design the fallback so user facing requests do not fail, while still preserving the prompt version audit trail when Langfuse comes back.
Cache prompts in process with a TTL, and fall back to a baked-in default on cache miss. Tag every trace with the version id that actually ran, and add a circuit breaker on the fetch path.
Think of how a restaurant handles its supplier going offline. The kitchen keeps a small stock of every ingredient in the back fridge (the cache), so a 30-minute supplier outage is invisible to diners. If even the back fridge is empty, the chef has a default recipe that uses pantry staples (the baked-in default) so service never stops. The receipt for each meal records which ingredient batch was actually used, including a tag like 'pantry-default' if it came from the fallback. When the supplier comes back, the audit trail explains exactly what was served during the outage, even though no diner ever saw a problem.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The principle that drives this design is simple and strict: the user-facing request must never block on the observability backend. Langfuse is a soft dependency, and a 30-second outage in a soft dependency should never become a 30-second outage in the product.
The right design has three layers that compose. An SDK-level cache with a TTL absorbs short outages invisibly. A baked-in default prompt provides the final fallback for cold-start during a long outage. An audit-trail discipline tags every trace with the prompt version that actually ran, so the post-incident review can reconstruct exactly what users saw. A circuit breaker on the fetch path prevents the recovery from being worse than the outage.
Mental model: the user path reads from the cache (or the baked default). The audit path writes the version id of whatever was actually used. The two paths are independent, so the audit survives even when fetches fail.
Layer 1: SDK cache with TTL
Every modern observability SDK that serves prompts exposes an in-process cache with a configurable TTL. Langfuse's get_prompt(name, cache_ttl_seconds=60) is the canonical example.
How the cache behaves
On first call for a (name, label) pair, the SDK fetches from Langfuse. It stores the prompt object in an in-process map keyed on (name, label). Subsequent calls within the TTL window return the cached object instantly. After the TTL expires, the next call still returns the cached object at once, and the SDK refreshes it in the background.
How the cache absorbs an outage
If the outage is 30 seconds, any process that has fetched the prompt before keeps serving it from cache. When the TTL expires during the outage, the background refresh fails and the cached prompt stays in use. In a healthy production system, the same prompts are fetched dozens of times per second. So almost every request is served from cache, and the outage is invisible.
Sizing the TTL
A warm cache hides an outage whatever the TTL, so the TTL mainly sets how fast intentional prompt updates propagate. Common production settings:
- 30 seconds: fast propagation.
- 60 to 120 seconds: typical choice.
- 5 to 10 minutes: fewer refresh calls, but needs an explicit cache-bust API for fast rollouts.
Cache-bust escape hatch
Most teams pair the TTL with an admin endpoint that forces an immediate refresh. This lets you push an emergency prompt change in seconds rather than waiting for the TTL to expire.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Langfuse Python and JS SDKs both expose cache_ttl_seconds on get_prompt() as the primary defense against fetch failures.
- LangSmith Hub prompts support similar in-process caching plus a fetch_with_default pattern for the same reason.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you size the TTL?
Pick the longest acceptable propagation delay for intentional prompt updates. Common values are 30 to 120 seconds. Pair with an explicit cache-bust endpoint for emergency rollouts when 60 seconds is too slow.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blocking the user request on Langfuse availability. A 30-second outage in the observability backend becomes a 30-second outage on every customer-facing call. That is the inverse of what observability is supposed to do.
60 second bullets to scan on the way to the call.
Why the user-facing path must not block on observability availability
How an SDK-level cache with TTL absorbs short outages
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.