Estimate the worker pool a summarization queue needs using Little's Law
A document summarization service drains jobs from a queue. It must sustain 7,200 jobs per hour, and each job occupies a worker for 20 seconds end to end (LLM latency plus I/O). Using Little's Law, how many concurrent workers are needed at minimum, and how many would you provision with 25 percent headroom?
Little's Law says L equals lambda times W. 2 jobs per second times 20 seconds equals 40 workers at zero slack; add 25 percent headroom to 50 workers for bursts and spikes.
Imagine a coffee shop where each customer takes twenty seconds at the counter, and two new customers walk in every second. To keep the line from growing forever, you need exactly forty baristas working at the same time. If you have less, the line grows. If you have exactly forty, you keep pace but any little bump (a slow customer, a barista sneezing) sends the line growing. So in real life you hire fifty: ten extra to absorb the bumps. The line still exists, but it stays short instead of stretching out the door.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Little's Law is one of those rare results that is simple, provable, and immediately useful. It says that in any stable queueing system, the average number of items in the system equals the arrival rate times the average time each item spends in the system. It makes no assumption about arrival distribution, service distribution, or scheduling discipline. It always holds.
Here it sizes a standard backend worker pool. The arithmetic is mechanical: convert units, multiply, add headroom. The interesting work is in understanding what the law tells you and what it does not. Then you bridge the gap between the analytic floor and a defensible production plan.
Stating and applying Little's Law
Little's Law in symbols:
where L is the average number of items in the system, lambda is the average arrival rate, and W is the average time each item spends in the system. The law applies to any stable queueing system, regardless of arrival or service distribution.
Here, lambda is 7,200 jobs per hour, which is 2 jobs per second. W is 20 seconds per job. So L is 40 jobs in flight on average at steady state.
In a worker-pool architecture, L corresponds to concurrent workers. If you have fewer than 40, the in-service count cannot reach 40. The system needs 40 to drain the arrival rate, so the queue grows without bound. If you have exactly 40, the system is at 100 percent utilization on average, which means any positive variance pushes the queue up.
The unit conversion matters. A common error is to leave arrival rate in jobs per hour and multiply by service time in seconds, getting a nonsensical number. Convert to consistent units (both in seconds, or both in hours) before multiplying.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Little's Law is the standard way to size consumer pools for queues such as AWS SQS and Azure Service Bus
- Celery, Sidekiq, and BullMQ documentation walks through this exact arithmetic for worker provisioning
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does this change if 5 percent of jobs take 200 seconds instead of 20?
Compute weighted mean service time: 0.95 x 20 + 0.05 x 200 = 19 + 10 = 29 seconds. New L is 2 x 29 = 58 workers. The tail dominates service time even at low frequency, which is why ignoring variance under-provisions.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing queue depth with worker count. The queue can be deep or shallow. What matters for throughput is the in-flight concurrency, which Little's Law fixes at 40.
60 second bullets to scan on the way to the call.
Statement of Little's Law in words and symbols
Unit conversion from per-hour to per-second
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.