Estimate the worker pool a summarization queue needs using Little's Law
A document summarization service drains jobs from a queue. It must sustain 7,200 jobs per hour, and each job occupies a worker for 20 seconds end to end (LLM latency plus I/O). Using Little's Law, how many concurrent workers are needed at minimum, and how many would you provision with 25 percent headroom?
Little's Law says L equals lambda times W. 2 jobs per second times 20 seconds equals 40 workers at zero slack; add 25 percent headroom to 50 workers for bursts and spikes.
Imagine a coffee shop where each customer takes twenty seconds at the counter, and two new customers walk in every second. To keep the line from growing forever, you need exactly forty baristas working at the same time. If you have less, the line grows. If you have exactly forty, you keep pace but any little bump (a slow customer, a barista sneezing) sends the line growing. So in real life you hire fifty: ten extra to soak up the bumps. The line still exists, but it stays short instead of stretching out the door.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Little's Law is one of those rare results that is simple, provable, and immediately useful. It says that in any stable queueing system, the average number of items in the system equals the arrival rate times the average time each item spends in the system. No assumption about arrival distribution, no assumption about service distribution, no assumption about scheduling discipline. It just holds.
This question applies it to a standard backend worker pool sizing problem. The arithmetic is mechanical: convert units, multiply, add headroom. The interesting work is in understanding what the law tells you, what it does not, and how to bridge the gap between the analytic floor and a defensible production plan.
This deep dive walks the arithmetic, the headroom decision, the variance and tail considerations, and the production discipline that pairs static sizing with autoscaling.
Stating and applying Little's Law
Little's Law in symbols:
where L is the average number of items in the system, lambda is the average arrival rate, and W is the average time each item spends in the system. The law applies to any stable queueing system, regardless of arrival or service distribution.
For this problem, lambda is 7,200 jobs per hour, which is 2 jobs per second. W is 20 seconds per job. So L is 40 jobs in flight on average at steady state.
In a worker-pool architecture, L corresponds to concurrent workers. If you have fewer than 40, the in-system count cannot reach 40, and since the system requires 40 to drain the arrival rate, the queue grows without bound. If you have exactly 40, the system is at 100 percent utilization on average, which means any positive variance pushes the queue up.
The unit conversion matters. A common error is to leave arrival rate in jobs per hour and multiply by service time in seconds, getting a nonsensical number. Convert to consistent units (both in seconds, or both in hours) before multiplying.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- AWS SQS and Azure Service Bus capacity planning guides explicitly use Little's Law to size consumer pools
- Celery, Sidekiq, and BullMQ documentation walks through this exact arithmetic for worker provisioning
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does this change if 5 percent of jobs take 200 seconds instead of 20?
Compute weighted mean service time: 0.95 x 20 + 0.05 x 200 = 19 + 10 = 29 seconds. New L is 2 x 29 = 58 workers. The tail dominates service time even at low frequency, which is why ignoring variance under-provisions.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing queue depth with worker count. The queue can be deep or shallow; what matters for throughput is the in-flight concurrency, which Little's Law fixes at 40.
60 second bullets to scan on the way to the call.
Statement of Little's Law in words and symbols
Unit conversion from per-hour to per-second
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.