Describe the two factors that decide how painful an observability vendor migration will be
Two factors decide migration pain: the wire shape of your spans (OTel-portable vs vendor-proprietary) and where your prompts and datasets live (your repo as source of truth vs the vendor UI as source of truth).
Imagine packing for a move. If your stuff is in standard cardboard boxes that fit any truck, moving day is easy: you call a new truck and load. If your stuff is in custom-shaped containers that only one truck accepts, you have to repack everything before you can move. Now imagine some of your stuff (recipes, photo albums, addresses) only lives at the old house, not in any box. You have to copy each one by hand to the new house. Observability vendors are the same. If your spans speak the standard language (OpenTelemetry) and your prompts and tests live in your code repo, switching vendors is a few days of work. If your spans speak only the vendor's private language and your prompts only live in the vendor's website, switching is a months-long project of rewriting code and copying content.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Observability vendor migrations are not rare. The LLM observability category is young, the vendors are evolving fast, and the right choice two years ago is often not the right choice today. Teams that survive these migrations cheaply made specific decisions at vendor-selection time. Teams that suffer through them did not.
Two factors decide migration pain. Span wire shape is the larger engineering cost, and artifact source of truth is the less visible organizational cost. The repo as source of truth pattern reduces both.
Factor one: span wire shape
Code at every LLM call site emits spans. The instrumentation library decides the shape of those spans (the attribute names, the structure of nested events, the way streaming responses are recorded).
Two paths
- Vendor SDK path. You import the vendor's Python or TypeScript library and wrap LLM calls with their decorators or context managers. Attribute names are vendor-specific (langfuse.observation.type, langsmith.run_type, helicone.cache_key). The shape is whatever the vendor decided.
- OpenTelemetry path. You instrument with OpenTelemetry plus a GenAI semantic convention library (OpenLLMetry from Traceloop, the native OTel tracing in the official Anthropic Python SDK, the OTel contrib instrumentation for the OpenAI SDK, or a hand-rolled OTel wrapper). Attribute names are standard (gen_ai.provider.name, gen_ai.usage.input_tokens, gen_ai.request.model). The shape is portable.
What migration costs in each path
Vendor SDK migration: rewrite every call site. A medium production system has forty to two hundred LLM call sites, with streaming, retry, and async variants. That is a calendar quarter of work, plus testing and rollout risk.
OTel migration: change the OTLP exporter URL in the OTel Collector configuration. It is one file, one line, one redeploy. The change itself takes an afternoon. The verification window (does the new vendor's UI show traces correctly) is a few days.
The order of magnitude difference is structural, not a matter of effort.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Setup | Span migration | Artifact migration | Total time |
|---|---|---|---|
| OTel + repo as source of truth | Change one URL | Re-point repo CI to new vendor | Days to one week |
| OTel + vendor UI for artifacts | Change one URL | Export/import prompts and datasets manually | 2-4 weeks |
| Vendor SDK + repo as source of truth | Rewrite all call sites | Re-point repo CI | 2-3 months |
| Vendor SDK + vendor UI for artifacts | Rewrite all call sites | Export/import + rebuild workflows | Calendar quarter |
Real products, models, and research that use this idea.
- Teams migrating from LangSmith to Langfuse who were already on OpenLLMetry plus OTel report changing one exporter URL. Teams on the LangSmith SDK with prompts in LangSmith Hub report multi-month projects.
- The Anthropic Python SDK ships native OpenTelemetry tracing, and the OTel contrib package opentelemetry-instrumentation-openai-v2 covers the OpenAI SDK. Both make OTel-shape the default if teams accept it.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through a concrete migration from LangSmith to Langfuse for a team that was already on OTel and had prompts in the repo.
Change OTLP exporter URL from smith.langchain.com to cloud.langfuse.com (or self-hosted endpoint), update auth header, re-point prompt-CI to the Langfuse prompts API. It takes under a week, mostly verification time.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking a vendor based on the dashboard demo without checking whether spans are OTel-portable and whether prompts and datasets can live in your repo. The lock-in shows up a year later, when migration becomes a calendar quarter of work.
60 second bullets to scan on the way to the call.
What are the two factors that decide observability migration pain?
Why do OTel-shape spans turn migration into a Collector config change?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.