List the selection criteria you would apply to a candidate LLM framework before adoption, and rank them by long term consequence
Your team is evaluating a candidate LLM framework for a multi-year product. List the selection criteria you would run it through before adoption, and rank them by long term consequence. Justify the ranking.
Rank by cost of being wrong: ejection path, observability, feature lag (architectural and irreversible) come before composition fit, language fit, streaming UX (tactical and adjustable).
Imagine choosing where to plant trees on a new property. Some choices are hard to undo. The root system grows wide, the trunk grows thick, and ten years in you cannot just move the tree. Other choices, like which annual flowers to plant, you can change every spring. Framing framework selection the same way: which decisions root deep and which can be redone next sprint. The ones that root deep need the most thought up front because the cost of being wrong is years of debt. Ejection cost is the deepest root. Streaming UX is the annual flower.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Framework selection for a multi-year product is one of the few engineering decisions where the wrong choice silently grows expensive for years before the bill comes due. The criteria a team weighs at adoption time and the criteria that matter at year three are usually different lists. The senior version of this question is not 'what criteria would you list', every candidate can list six, but 'how do you rank them, and why does the ranking invert what most teams do'.
This dive walks through the six criteria, the principle that ranks them by cost of being wrong rather than frequency of mattering, and the architectural patterns that mitigate the top three even when the choice itself was imperfect.
Why cost of being wrong beats frequency of mattering
The instinct when ranking criteria is to weight by daily importance. Composition fit is what your engineers touch every day; language ergonomics decides whether the codebase feels good to write; streaming UX shapes the product's chat surface. These are the criteria that feel important, and most teams rank by them.
The inversion: those are the cheap criteria. If your composition altitude is wrong, you fix it at the sub-feature granularity. Wrap LangGraph around your LCEL chain, drop into raw SDK for the hot loop. If language ergonomics is bad, you build a thin TypeScript layer over the Python framework, or vice versa. If streaming UX is wrong, you swap the streaming primitive per feature. All within-sprint adjustments.
The expensive criteria are the ones that decide whether you can change your mind. Ejection cost decides whether you can switch frameworks at all. Observability compatibility decides which vendor you are committed to. Feature-lag tolerance decides whether your product can ship the next provider launch on day one. These criteria are architectural; their cost of being wrong is multi-quarter, not multi-sprint, and the mitigations have to be designed in at adoption time, not retrofitted later.
Ranking by cost of being wrong shifts the top three to ejection, observability, and feature lag. And that ranking matches what production teams discover after their first migration.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Criterion | Tier | Cost of being wrong | Mitigation |
|---|---|---|---|
| Ejection path | Architectural | Multi-quarter migration | Wrap behind own interfaces |
| Observability compatibility | Architectural | Vendor pinned for years | Pick OTel-conformant framework |
| Feature-lag tolerance | Architectural | Structurally behind the curve | Keep SDK-direct code paths |
| Composition complexity | Tactical | Dead weight or hand-rolling | Re-pick at sub-feature granularity |
| Team language fit | Tactical | Velocity tax | Pick TS-first for TS teams |
| Streaming UX | Tactical | Worse user experience | Switch streaming primitive per feature |
Real products, models, and research that use this idea.
- Notion's framework selection process documented their move to wrap LangChain behind internal interfaces to keep ejection cost bounded.
- Stripe engineering picked Langfuse over LangSmith partly because OTel-conformant traces kept the framework decision independent of the observability decision.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you write the integration test that proves ejection cost is bounded?
Define your own Generator / Retriever / Parser interfaces. Implement them with the framework. Write a stub implementation that returns canned responses. Run the same test suite against both implementations and assert business logic passes either way. If it does, the boundary holds and ejection is one configuration change.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Ranking by 'how often the criterion matters in daily work' (which puts composition fit at the top) instead of by 'how expensive the wrong decision becomes over time' (which puts ejection cost first).
60 second bullets to scan on the way to the call.
The cost of being wrong ranking principle
Why ejection cost is the load-bearing criterion
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.