List the selection criteria you would apply to a candidate LLM framework before adoption, and rank them by long term consequence
Your team is evaluating a candidate LLM framework for a multi-year product. List the selection criteria you would run it through before adoption, and rank them by long term consequence. Justify the ranking.
Rank by cost of being wrong: ejection path, observability, feature lag (architectural and irreversible) come before composition fit, language fit, streaming UX (tactical and adjustable).
Imagine choosing where to plant trees on a new property. Some choices are hard to undo. The root system grows wide, the trunk grows thick, and ten years in you cannot just move the tree. Other choices, like which annual flowers to plant, you can change every spring. Framework selection works the same way: some decisions root deep and others can be redone next sprint. The ones that root deep need the most thought up front because the cost of being wrong is years of debt. Ejection cost is the deepest root. Streaming UX is the annual flower.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Framework selection for a multi-year product is one of the few engineering decisions where the wrong choice silently grows expensive for years before the cost becomes visible. The criteria a team weighs at adoption time and the criteria that matter at year three are usually different lists. Listing six criteria is easy. The harder part is ranking them, and seeing why the ranking inverts what most teams do.
The ranking principle is cost of being wrong rather than frequency of mattering. Architectural patterns can mitigate the top three criteria even when the choice itself was imperfect.
Why cost of being wrong beats frequency of mattering
The instinct when ranking criteria is to weight by daily importance. Composition fit is what your engineers touch every day. Language ergonomics decides whether the codebase feels good to write. Streaming UX shapes the product's chat surface. These are the criteria that feel important, and most teams rank by them.
The inversion: those are the cheap criteria. If your composition altitude is wrong, you fix it at the sub-feature granularity. Wrap LangGraph around your LCEL chain, or drop into raw SDK for the hot loop. If language ergonomics is bad, you build a thin TypeScript layer over the Python framework, or vice versa. If streaming UX is wrong, you swap the streaming primitive per feature. All of these are within-sprint adjustments.
The expensive criteria are the ones that decide whether you can change your mind. Ejection cost decides whether you can switch frameworks at all. Observability compatibility decides which vendor you are committed to. Feature-lag tolerance decides whether your product can ship the next provider launch on day one. These criteria are architectural. Their cost of being wrong is multi-quarter, not multi-sprint. The mitigations have to be designed in at adoption time, not retrofitted later.
Ranking by cost of being wrong shifts the top three to ejection, observability, and feature lag. That ranking matches what production teams discover after their first migration.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Criterion | Tier | Cost of being wrong | Mitigation |
|---|---|---|---|
| Ejection path | Architectural | Multi-quarter migration | Wrap behind own interfaces |
| Observability compatibility | Architectural | Vendor pinned for years | Pick OTel-conformant framework |
| Feature-lag tolerance | Architectural | Behind on new provider features | Keep SDK-direct code paths |
| Composition complexity | Tactical | Dead weight or hand-rolling | Re-pick at sub-feature granularity |
| Team language fit | Tactical | Slower delivery | Pick TS-first for TS teams |
| Streaming UX | Tactical | Worse user experience | Switch streaming primitive per feature |
Real products, models, and research that use this idea.
- Teams that wrap LangChain behind internal interfaces keep ejection cost bounded, because business logic never imports framework types.
- Teams that pick Langfuse with OTel-conformant traces keep the framework decision independent of the observability decision.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you write the integration test that proves ejection cost is bounded?
Define your own Generator / Retriever / Parser interfaces. Implement them with the framework. Write a stub implementation that returns canned responses. Run the same test suite against both implementations and assert business logic passes either way. If it does, the boundary holds and ejection is one configuration change.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Ranking by 'how often the criterion matters in daily work' (which puts composition fit at the top) instead of by 'how expensive the wrong decision becomes over time' (which puts ejection cost first).
60 second bullets to scan on the way to the call.
The cost of being wrong ranking principle
Why ejection cost is the most important criterion
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.