Pick the isolation controls a multi-tenant LLM API must enforce
Pick the three per-tenant controls: rate limits, cost caps, and scoped context/cache/logs. A global limit and a query-only shared cache both fail to isolate customers.
Imagine an apartment building where the word 'per-apartment' decides everything. Each unit needs its own water limit so one resident can't drain the tank, its own utility meter so nobody runs up a shared bill, and its own locked mailbox so mail never lands in the wrong slot. A single meter for the whole building doesn't tell you who used what, and one shared mailbox means everyone reads everyone's letters. The right answers all start with 'per-tenant'; the wrong ones lump everyone together or share something that should be private.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Spend 5-7 minutes on the per-tenant test and on why the two distractors are genuinely tempting but break isolation in different ways.
Real products, models, and research that use this idea.
- LLM gateways like Portkey and LiteLLM apply per-tenant (per-API-key) rate limits and token budgets rather than a single global ceiling.
- Vector databases such as Pinecone use per-tenant namespaces so a retrieval can't return another tenant's documents.
- Production semantic caches add tenant ID to the cache key to keep the cost savings without the cross-tenant leak.
What an interviewer would ask next. Try answering before peeking at the approach.
QCould you ever safely share a semantic cache across tenants?
QWhen is a global rate limit still useful even though it doesn't isolate tenants?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Selecting a single global rate limit as isolation — it caps total load but lets one noisy tenant consume the whole budget and starve every other customer.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.