Design an LLM code review bot for a 500 engineer organization. Roughly 400 pull requests land per day, arriving in bursts after standups. Reviews must post within a few minutes of a push, the bot must never block merges, and the platform team has a hard monthly budget. Walk through the architecture and your main cost and latency levers.
Webhook into queue, worker pool drains, diff-scoped chunks, model routed by stakes, per-file cache on content hash, non-blocking checks, hard token caps with a per-repo spend dashboard.
Think of a kitchen during the lunch rush. Orders (pull requests) come in a wave at the same time. If the kitchen tried to cook every order in parallel the moment it arrived, the gas line would blow. Instead, there is a queue printer. Cooks pull tickets one at a time. Simple sandwiches go to the apprentice; the steak special goes to the head chef. The kitchen never refuses to serve a customer because the printer jammed: if the bot is down, the customer still gets their food, just without the chef's note. And the manager watches the spend at the end of every day, not at the end of the month.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A code review bot is one of the cleaner LLM system design problems because the constraints are explicit and unforgiving. Four hundred PRs a day in standup-shaped bursts, a minutes not seconds latency promise, never block a merge, hard monthly budget. Those four constraints determine most of the architecture, and the rest comes from choosing the right knob on each axis.
This deep dive walks the system from webhook to comment, then steps back and names the four levers that decide whether the bot is operationally sane or a recurring incident generator: diff scoping, routing, caching, and the merge-safety invariant.
Ingestion, queue, and worker pool
The first decision is decoupling. PR webhooks fire whenever a developer pushes, and the receiver has tight response-time requirements (webhook providers typically time out within seconds and may disable the integration after repeated failures). A receiver that calls the model inline is one bad latency day away from being silently disabled.
The shape that survives is a thin HTTP receiver that does three things: verify the webhook signature, deduplicate against an event-ID store, and enqueue a job. The receiver responds in milliseconds. A worker pool drains the queue concurrently with per-repo limits so one noisy repo cannot starve the others. Provider rate limits become a knob on pool size: if the provider tier is two thousand requests per minute and review jobs average ten seconds, the steady-state worker count is around three hundred and thirty before any limit binds.
The queue is also the burst absorber. Four hundred PRs a day is an average of roughly five per minute, but a standup at 10 a.m. across a few timezones can produce a thirty-PR minute. Without a queue, that minute becomes a wall of provider 429s and a fraction of PRs reviewed; with a queue, the burst becomes a few seconds of additional queueing latency for the tail of the spike, well inside the minutes-scale SLO.
The operational pattern to wire from day one: queue depth and worker saturation are the two metrics that page on. When they trend upward together, the pool is undersized; when depth grows but saturation does not, a worker dependency is stuck.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
from fastapi import FastAPI, Request, BackgroundTasks
import hmac, hashlib, os
from queue import Queue
app = FastAPI()
job_queue: Queue = Queue()
@app.post('/webhook')
async def gh_webhook(req: Request, bg: BackgroundTasks):
body = await req.body()
sig = req.headers.get('X-Hub-Signature-256', '')
mac = hmac.new(os.environ['GH_SECRET'].encode(), body, hashlib.sha256).hexdigest()
if not hmac.compare_digest(f'sha256={mac}', sig):
return {'ok': False}, 401
event = await req.json()
if event.get('action') in {'opened', 'synchronize'}:
job_queue.put({'pr': event['pull_request']['url'], 'id': event['delivery']})
return {'ok': True}Real products, models, and research that use this idea.
- GitHub Copilot for Pull Requests posts AI summaries and review comments as non-blocking checks, which is the exact merge-safety posture this design follows.
- CodeRabbit and Greptile both expose path-sensitivity rules and per-file caching to keep cost under control on large monorepos.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you set the per-PR token cap?
Start from the monthly budget divided by expected PR volume, with headroom; tune from observed p95 token-per-PR after a month of data.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Sending whole files instead of changed hunks, blocking merges on bot latency, using one model for every PR regardless of stakes, or having no token cap so one giant PR consumes a week of budget.
60 second bullets to scan on the way to the call.
Why does this architecture need a queue between the webhook and the worker?
What input scoping decisions cut model spend without losing review quality?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.