Pick the isolation controls a multi-tenant LLM API must enforce

Attempt it

Quick reference

TL;DR

Pick the three per-tenant controls: rate limits, cost caps, and scoped context/cache/logs. A global limit and a query-only shared cache both fail to isolate customers.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine an apartment building where the word 'per-apartment' decides everything. Each unit needs its own water limit so one resident can't drain the tank, its own utility meter so nobody runs up a shared bill, and its own locked mailbox so mail never lands in the wrong slot. A single meter for the whole building doesn't tell you who used what, and one shared mailbox means everyone reads everyone's letters. The right answers all start with 'per-tenant'; the wrong ones lump everyone together or share something that should be private.

Key concepts

LLM System Design Context Engineering Retrieval-Augmented Generation

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Spend 5-7 minutes on the per-tenant test and on why the two distractors are genuinely tempting but break isolation in different ways.

Real-world examples

Real products, models, and research that use this idea.

LLM gateways like Portkey and LiteLLM apply per-tenant (per-API-key) rate limits and token budgets rather than a single global ceiling.
Vector databases such as Pinecone use per-tenant namespaces so a retrieval can't return another tenant's documents.
Production semantic caches add tenant ID to the cache key to keep the cost savings without the cross-tenant leak.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QCould you ever safely share a semantic cache across tenants?

Sign in or go Pro to see the approach for this follow-up.

QWhen is a global rate limit still useful even though it doesn't isolate tenants?

Sign in or go Pro to see the approach for this follow-up.

QHow would you isolate cost when many tenants share one cached system-prompt prefix?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Selecting a single global rate limit as isolation — it caps total load but lets one noisy tenant consume the whole budget and starve every other customer.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

Multitenancy (overview)