Design a support copilot for a SaaS company handling 1,000,000 tickets per month. It drafts replies for human agents, must ground answers in help center docs and resolved past tickets, keep p95 draft latency under 8 seconds, and stay under $0.02 per ticket. Walk through the end to end architecture and the tradeoffs you would call out.
A million tickets a month is barely 1 QPS, so design for unit economics and quality: hybrid retrieval, tiered model routing, prompt and response caching, streamed drafts, and agent edits as the feedback loop.
Picture a new assistant at a busy help desk. The assistant never talks to customers directly; instead, it writes a suggested reply on a sticky note and hands it to an experienced staff member, who fixes it or throws it away. Before writing anything, the assistant flips through two binders: the official manual and a folder of answers that worked before. Most questions are routine, so a quick junior assistant handles them; the rare tricky ones go to the senior expert, who costs more per answer. Common questions get a pre-written card so nobody drafts the same reply twice. And every correction the staff member makes is a lesson: the more they edit, the more the team learns where the assistant is weak.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question is a disguised unit economics problem wearing a system design costume. Candidates who treat it as a scaling exercise spend their time on load balancers nobody needs; candidates who do thirty seconds of arithmetic discover the real game, which is shipping good drafts at two cents each while the underlying docs keep changing.
The walkthrough below follows the order a strong interview answer should: size the load, build the retrieval foundation, design generation and routing, attack the cost target with numbers, meet the latency budget, and close the quality loop that human agents provide for free.
Size the load before designing anything
A million tickets a month sounds like infrastructure. Divide it out:
Support traffic concentrates in business hours, so assume a 10x to 25x peak factor: 4 to 10 drafting calls per second at the worst moment. Every major provider serves that from a default rate tier. There is no sharding story here, no GPU fleet, no queueing theory emergency.
What the math does is reallocate your design attention. At 1 QPS, a 100 millisecond inefficiency is invisible; a one cent inefficiency is $10,000 a month. The budget line, $0.02 times a million tickets, is $20,000 a month of model spend, and finance will hold the line. Meanwhile quality is measured a million times a month by professional reviewers, your own support agents, who will simply stop reading drafts if the drafts waste their time.
So the constraints rank: cost per ticket first, draft quality second, latency third (8 seconds is generous), throughput a distant fourth. Saying this ordering out loud, with the arithmetic that justifies it, is the strongest possible opening because it shows you derive designs from numbers rather than pattern-matching to big system templates.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Cost lever | What it cuts | Typical effect |
|---|---|---|
| Prompt caching | Input cost of the static system prompt and policies | Up to 90 percent off the cached prefix |
| Tight retrieval | Input tokens from context chunks | 3 to 5 chunks instead of 20 cuts most input spend |
| Response caching | Entire calls for repeated intents | Near zero marginal cost on cache hits |
| Output caps | Output tokens, the priciest line | Bounded worst-case cost per draft |
| Tiered routing | Premium model spend | Strong model only on the hard minority |
Real products, models, and research that use this idea.
- Intercom Fin grounds answers in the customer's help center and hands off to human agents, the same retrieval plus human gate shape at production scale.
- Decagon and Sierra both sell enterprise support agents built around grounded generation with human escalation paths and per-resolution pricing pressure.
What an interviewer would ask next. Try answering before peeking at the approach.
QAcceptance rate is 85 percent but customer satisfaction is flat. What do you investigate?
Consider rubber-stamping agents, edit distance as the truer signal, and slicing acceptance by intent and retrieval confidence.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Jumping straight to sharding and GPU clusters. At under one call per second average, throughput is trivial; the binding constraints are the two cent budget and draft quality.
60 second bullets to scan on the way to the call.
What does the QPS math say about where the real constraints are in this design?
Why hybrid retrieval over docs plus resolved tickets, and what keeps both sources fresh?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.