- 1API accepts the request and validates it
- 2Notify the caller via webhook (or let them poll the job ID)
- 3A worker pulls the job from the queue and calls the model
- 4Enqueue a job and immediately return a job ID to the caller
- 5Persist the result and update the job's status to done
Accept and validate, enqueue and return a job ID immediately, let a worker call the model, persist the result, then notify via webhook or polling.
Imagine dropping film off at a photo shop. You don't stand at the counter while they develop it — they hand you a ticket and say come back later. The shop processes your film in the back when a worker is free. A long LLM job works the same way. The API takes your request, gives you a ticket (the job ID), and a worker in the back calls the model. When it's done, the shop calls you or you check your ticket. You were never stuck waiting at the counter.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Long-running LLM work breaks the assumption every web API is built on: that a request returns quickly. A document summarizer chewing through 50 pages, a multi-step agent making a dozen tool calls, a batch of 10,000 classifications — these take seconds to minutes, far past any sane HTTP timeout. Hold the connection open and load balancers, proxies, and browsers all give up on you.
The answer is the same pattern backend engineers have used for slow work for decades: a job queue. But applying it to LLM systems has its own wrinkles — model calls hang, they cost real money per attempt, and they fail nondeterministically — so the ordering and the durability story matter more than usual.
This deep dive walks the five steps in order, explains why each one sits where it does, and shows the two reorderings that quietly rebuild the synchronous design you were trying to escape. The goal is that you can defend not just the list but the dependencies between its steps.
Why the synchronous path collapses under slow model calls
Start with the failure you're designing against. A normal request-response API works because the handler does a little work and returns. Wire an LLM call that takes 90 seconds into that handler and several things break at once.
The client times out. Browsers, mobile clients, and most HTTP libraries abandon a request after 30-60 seconds. Reverse proxies and load balancers cut idle connections even sooner. So the user sees an error even though the model is still working — and you've already paid for the tokens.
Worse, every in-flight slow request holds a server thread or connection hostage. Under load, a burst of long jobs exhausts your web tier's capacity and starves the fast requests sharing it. One slow endpoint takes down the whole service.
The core problem is coupling: user-facing latency is chained to model latency. The async pattern exists to break that chain. Once you see the failure clearly, every step in the pipeline is just a consequence of decoupling the fast accept path from the slow work path.
This is the same reasoning that pushed every other slow operation — video encoding, report generation, bulk email — off the request path decades ago. LLM jobs just make the problem acute, because a single model call can run longer than an entire traditional request, and it can hang or fail in ways a database query rarely does.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# API tier: accept, enqueue, return immediately
@app.post("/summarize")
def summarize(req: SummarizeReq):
validate(req) # 1. fail fast on bad input
job_id = uuid4().hex
db.create_job(job_id, status="queued")
queue.enqueue("run_summary", job_id, req.doc) # 2. enqueue
return {"job_id": job_id} # 2b. return ID BEFORE any model call
# Worker tier: dequeue, call model, persist, notify
def run_summary(job_id, doc):
db.set_status(job_id, "running")
result = llm.summarize(doc, timeout=120) # 3. slow model call
db.save_result(job_id, result) # 4. persist
db.set_status(job_id, "done")
fire_webhook(job_id) # 5. notify (idempotent)Real products, models, and research that use this idea.
- Celery with Redis or RabbitMQ is the classic Python job-queue setup teams use to offload long LLM summarization and agent runs.
- AWS SQS plus Lambda workers, with results in DynamoDB and status polled by job ID, is a common serverless async-LLM pattern.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you make the worker safe to retry if it crashes after calling the model but before persisting the result?
Discuss idempotency keys on the job, visibility timeouts so the message reappears, and detecting whether the model call already succeeded to avoid double-charging.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Having the API call the model inline and only return a job ID afterward — which defeats the entire point of going async and reintroduces the timeout.
60 second bullets to scan on the way to the call.
Why a slow LLM call cannot live on the synchronous request path
What the API must return before any model call happens
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.