Hosting model, OTel-native posture, eval depth, framework coupling, and pricing scaling. Logo design is not a real axis.
Picture choosing a delivery service for a small business. The real questions are: do they deliver in our area, do they handle the volume we send, do they fit our tools, what does it cost per parcel, and can we switch later without losing our address book. Whether their truck has a cute paint job is irrelevant to the decision. Choosing an LLM observability vendor is the same. Five concrete axes carry actual decision weight. Treat the marketing-flavored stuff as background noise.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM observability vendor selection is a recurring decision: teams pick once, regret part of the choice within a year, and either migrate or live with it. Picking well the first time means understanding which axes carry real decision weight and which are marketing noise.
This question is straightforward as multi-select but is really an exercise in calibrated trade-off thinking. The five axes are the substrate of every vendor comparison conversation in the space.
Self-host vs hosted: the deepest axis
This axis decides where your prompt and output text physically lives. For some teams it is a soft preference; for others it is a hard line.
Self-hosted options. Langfuse, Phoenix, OpenLLMetry plus a self-hosted OTel collector, TruLens. Trace data stays in your VPC or on-prem. Compliance posture is simpler (no third-party data processor for the trace content). Operating cost is real: you run the four-store stack discussed in the Langfuse self-host question, plus monitoring, plus backups, plus upgrades.
Hosted options. LangSmith, Braintrust, Datadog LLM Observability, Helicone, Weights & Biases Weave. Zero ops. Prompt and output text leaves your perimeter, which puts the vendor on your data processor list and may trigger contract review with security and legal.
For regulated industries (healthcare under HIPAA, finance under PCI/SOC controls, EU under the AI Act and GDPR), hosted vendors are often off the table for prompt and output content. Some teams compromise: hosted with strict PII redaction before egress, plus contractual data-processing agreements.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Langfuse 3.x: self-hostable, OTel-native, deep eval features, framework-agnostic, strong on four of five axes.
- LangSmith: hosted, partial OTel ingest, deep eval features, tight LangChain coupling, strong fit if you are already in the LangChain ecosystem.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build a scoring matrix that turns these axes into a concrete recommendation?
Assign weights per axis based on the team's constraints, score each vendor 1-5 per axis, multiply and sum.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Selecting on feature-list maximalism. The right vendor matches your specific constraints, not the longest feature comparison table.
60 second bullets to scan on the way to the call.
Name the five real axes for LLM observability vendor selection
Identify which axis tends to dominate for regulated-industry teams
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.