Pick the isolation controls a multi-tenant LLM API must enforce
Pick the three per-tenant controls: rate limits, cost caps, and scoped context/cache/logs. A global limit and a query-only shared cache both fail to isolate customers.
Imagine an apartment building where the word 'per-apartment' decides everything. Each unit needs its own water limit so one resident can't drain the tank, its own utility meter so nobody runs up a shared bill, and its own locked mailbox so mail never lands in the wrong slot. A single meter for the whole building doesn't tell you who used what, and one shared mailbox means everyone reads everyone's letters. The right answers all start with 'per-tenant'; the wrong ones lump everyone together or share something that should be private.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question looks like a checklist, but it's really testing one mental model: what does it mean to isolate one customer from another on shared infrastructure? Get that model right and the five options sort themselves instantly.
The trap is that every option sounds like good engineering. Rate limits are good. Cost caps are good. Caches save money. A global limit protects you from overload. So a candidate skimming for 'reasonable-sounding controls' picks the wrong set. The discriminator isn't whether a control is useful in general — it's whether it draws a boundary around the individual tenant.
This deep dive builds that discriminator. We'll establish the per-tenant test, apply it to the three correct controls, then dissect why the two distractors are tempting and exactly how each one fails. The goal is that you leave able to look at any proposed control and answer in one beat: does this isolate per tenant, or does it lump everyone together?
The per-tenant test that sorts every option
Isolation has a precise meaning: no tenant's behavior or data can affect another tenant. That definition gives you a one-question test for any control — is it scoped to the individual tenant, or to the whole pool?
Run the test on the three correct options and they pass cleanly. A per-tenant rate limit gives each customer its own bucket, so customer A's burst can't drain customer B's allowance. A per-tenant cost cap gives each customer its own budget, so A's runaway loop can't spend B's money or hide inside an aggregate total. Tenant-scoped context and caches give each customer its own data partition, so A's documents can't surface in B's answers.
Now run it on the distractors. A single global rate limit is scoped to the pool, not the tenant — it fails. A semantic cache keyed only on query text is scoped to the query, not the tenant — it also fails, because the tenant boundary is simply absent from the key.
The test is mechanical and fast. You don't have to reason about each control's mechanism in depth; you just ask where the boundary is drawn. If the answer isn't 'around one tenant', it isn't isolation, no matter how useful the control is for something else.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LLM gateways like Portkey and LiteLLM apply per-tenant (per-API-key) rate limits and token budgets rather than a single global ceiling.
- Vector databases such as Pinecone use per-tenant namespaces so a retrieval can't return another tenant's documents.
What an interviewer would ask next. Try answering before peeking at the approach.
QCould you ever safely share a semantic cache across tenants?
Only for entries derived from tenant-agnostic public content; any entry shaped by tenant-specific input must be keyed or partitioned by tenant ID.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Selecting a single global rate limit as isolation — it caps total load but lets one noisy tenant consume the whole budget and starve every other customer.
60 second bullets to scan on the way to the call.
Why 'per-tenant' is the test that sorts isolation controls from non-isolation ones
The three threats each correct control addresses
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.