1. System Performance: Queuing

2. Queuing Theory
Leonard Kleinrock - “We study the phenomena of standing, waiting, and serving, and we call this study Queuing Theory. Any system in which arrivals place demands upon a finite capacity resource may be termed a queuing system.”

3. Airport Security Waiting Line

4. Common Queuing Questions
How long will it take to be served?
What length of line will build up?
How long a line is desirable/tolerable?
How many servers do I need?
What throughput can I expect?

5. Queuing Systems Examples Banks Grocery stores Highway traffic
Emergency rooms
Communications networks
Computers
Emergency call centers

6. Queuing Issues
Relative Economics
Capacity
System Architecture

7. Queuing Systems Processor(s) Queue(s) Transactions Events
Management Discipline
Transactions
Events
Arrivals
Departures

8. Queuing Systems

9. Queuing Algorithm Arrival Departure
If the server is idle, the transaction will go into the processor and will depart after it is serviced
If the server is busy, the arriving transaction goes into the queue
Departure
If the queue is empty, the processor goes idle
If the queue is non-empty, a transaction is selected from the queue based on the queue discipline and the transaction will go into the processor and will depart after it is serviced

10. Queuing Experience If traffic is light If traffic is heavy
Queues will be small
We get served quicker
If traffic is heavy
Queues will grow
It takes longer to get served

11. Queuing Theory Steady-state performance Mathematical model Systems
Networks
Mathematical model
Mean Value Analysis

12. Arrivals Exponential inter-arrival times Poisson Distribution
Probability of n arriving in the next t time intervals
l = mean (expected) arrival rate
Pn(t) = (lt)n/n! e-lt
Example:
Probability of 3 arriving in the next second if the mean arrival rate is 6 per second
P3(1) = (6*1)3/3! e-6*1
P3(1) =

13. Poisson Arrival Distribution

14. Example
An emergency supply dispensing station has an expected arrival rate of 5 people per minute to receive survival kits.
What is the probability of 10 arriving in the next minute?
What is the probability of 5 or more arriving in the next minute?

15. Example Solution A. Pn(t) = (lt)n/n! e-lt
P10(1) = (5*1)10/10! e-5*1
P10(1) = / *
P10(1) =
B. Pn>=l(t) = 1 – Sn-1i=0 Pi(t)
Pn>=5(t) = 1 – (P0(t) + P1(t) + P2(t) + P3(t) + P4(t))
Pn>=5(t) = 1 – ( )
Pn>=5(t) =

16. Example Observations Actual number of arrivals varies.
We need to look at the likelihood of many arrivals occurring
How fast must we process these arrivals to avoid long lines and delays?

17. Queuing Theory System description – Kendall Notation A/B/c/K/m/Z
A - arrival distribution
B - service time distribution
c - number of servers
K - system capacity
(i.e., the maximum number of jobs waiting in the queue, normally taken to be infinite)
m - source population
(normally taken to be infinite)
Z is the queuing discipline
(normally taken to be FIFO)

18. Kendall Notation Symbols
Meaning
Usage GI
General Independent arrival time A G
General service time B M
Markov (i.e. exponential) inter-arrival/service time
A/B D
Deterministic inter-arrival/service time
FIFO
First In/First Out queue discipline Z
LIFO
Last In/First Out queue discipline
RSS
Random Selection for Service queue discipline

19. Base Queuing Assumptions
M/M/1
Exponential arrival and service distributions
One server
M/D/1
Exponential arrival distribution
Deterministic (constant) service distribution
M/G/1
General service distribution
M/M/n
n servers

20. Queue Management Disciplines
FIFO – First-In-First-Out
(FCFS) First-Come-First-Served
LIFO – Last-In-First-Out
RSS – Random Selection for Service
Priority
Other

21. Queuing Model Parameters
Expected Processing Time a
Traffic Intensity c
Number of Processors 
Mean Arrival Rate 
Mean Processing Rate 
Average Processor Utilization L
Expected Number in the System Lq
Expected Number in the Queue W
Expected Wait Time in the System Wq
Expected Wait Time in the Queue g
Average Throughput

22. Queuing Model Formulas
W = Wq + S
Expected Wait Time in the System =
Expected Wait Time in the Queue +
Expected Processing Time
L = Lq + a
Expected Number in the System =
Expected Number in the Queue +
Traffic Intensity

23. Queuing Model Formulas
Little's Formula
L = W
Expected Number in the System =
Mean Arrival Rate * Expected Wait Time in the System
Lq = Wq

24. Queuing Model Formulas
a = l/
 = a/c
g = m * r

25. Queue State Transition Diagram

26. Queuing Model Development
At Steady State
pn-1 = pn
Where pn is the probability of the system state being n
Pn = l/m pn-1
Pn = an p0

27. Queuing Model Development
From probability
S pn = 1
S an p0 = p0 S an =1
This is a standard geometric series:
S xn =1/(1-x)

28. Queuing Model Development
p0 S an = p0 /(1 – a) = 1
p0 has a special interpretation:
The probability that zero is in the system
The system is idle
P0 = (1 – r)
Pn = r (1 - r)

29. Queuing Model Development
L = E(pn ) = r /(1 - r )
s2 = r /(1 - r )2

30. M/M/1 Formulas Pn(t) = (lt)n/n! e-lt S = 1/ a = l/  = a/c g = m * r
L = E(pn ) = r /(1 - r )
s2 = r /(1 - r )2
L = W
Lq = Wq
W = Wq + S
L = Lq + a

31. Queuing Example
A particular processor has information arriving from sensors over a communication line. The sensor information arrives with an exponential inter-arrival time that has a mean of 1000 sensor transactions per second. The processor processes these with an exponential service time with a mean of 1200 sensor transactions per second.

32. Example Calculations l 1000 m 1200 c 1 a 0.83333333 S 0.00083333 r L 5Lq W
0.005 Wq g

33. Queuing Example
Suppose that the mean arrival rate increases or decreases. How does this affect the M/M/1 example queuing model?

34. Variation of Mean Arrival Rate
100
300
600
900
1100 m
1200 c 1 a
0.25
0.5
0.75 S r L 3 11 Lq
2.25 W Wq

35. Queuing Example Observations
At low mean arrival rates, most of the time is spent in the processor.
At high mean arrival rates, most of the time is spent in the queue.
At higher utilizations, increases in the mean arrival rate result in disproportionate increases in mean system wait time.

36. Explosion of Expected Queue Size with Arrival Rate

37. Queuing Example L is the expected (mean) number in the system.
What are the probabilities of certain numbers being in the system?
Pn = r (1 - r )

38. Probability of n in the SystemPn
Cumulative Pn P0
0.1667 1 P1
0.1389
0.3056 2 P2
0.1157
0.4213 3 P3
0.0965
0.5177 4 P4
0.0804
0.5981 5 P5
0.0670
0.6651 6 P6
0.0558
0.7209 7 P7
0.0465
0.7674 8 P8
0.0388
0.8062 9 P9
0.0323
0.8385 10
P10
0.0269
0.8654 11
P11
0.0224
0.8878 12
P12
0.0187
0.9065 13
P13
0.0156
0.9221 14
P14
0.0130
0.9351 15
P15
0.0108
0.9459 16
P16
0.0090
0.9549 17
P17
0.0075
0.9624 18
P18
0.0063
0.9687 19
P19
0.0052
0.9739 20
P20
0.0043
0.9783

39. Probability of n in the System

40. Supply Distribution Example Revisited
An emergency supply dispensing station has an expected arrival rate of 5 people per minute to receive survival kits.
What length of queue and what time to service results if service is exponential and has a mean of 7 people per minute?

41. Example Calculations l 5 m 7 c 1 a
0.714 S
.1429 r L
2.5 Lq W Wq g

42. M/D/1 Queuing Model Development
Service is constant
L = E(pn ) = r /2(1 - r )
s2 = 0

43. Example: Emergency Inoculations
There are many examples where large numbers of people need to receive vaccinations, injections of antibiotics or other treatments. Medical personnel are trained to perform these actions methodically (e.g. wipe clean, inject, wipe, bandage, etc.). The result is that it requires the same amount of time to treat each patient. Suppose that a medical technician can treat 200 people per hour. If 150 people an hour arrive for treatment, what mean queue size will result and what will be the average service time?

44. Emergency Inoculations
150 m
200 c 1 a
0.75 S
0.005 r L
1.5 Lq
1.495 W
0.01 Wq

45. Queuing Example
Suppose that the mean arrival rate increases or decreases. How does this affect the M/D/1 example queuing model?

46. Variation of Mean Arrival Rate75
100
150
175
190 m
200 c 1 a
0.375
0.5
0.75
0.875
0.95 S
0.005 r L
0.3
1.5
3.5
9.5 Lq
0.295
0.495
1.495
3.495
9.495 W
0.004
0.01
0.02
0.05 Wq

47. Queuing Example Observations
At low mean arrival rates, most of the time is spent in the processor.
At high mean arrival rates, most of the time is spent in the queue.
At higher utilizations, increases in the mean arrival rate result in disproportionate increases in mean system wait time.

48. Explosion of Expected Queue Size with Arrival Rate

49. Queuing Example L is the expected (mean) number in the system.
What are the probabilities of certain numbers being in the system?
Pn = r (1 - r )

50. Comparison of M/M/1 and M/D/1 Queuing Models

51. M/G/1 Queuing Model General Lq = ((ls)2 + r2)/(2 *(1-r))
For Exponential
s = 1/m
For Deterministic
s = 0

52. Multiple Processor M/M/c Queues
Probability of having no items in queue
Probability of having n items in queue

53. M/M/c Queues
Probability of being in a queue

54. M/M/c Queues M/M/c Average length of queue
Average time waiting in queue
λ = arrival rate μ = departure rate

55. M/M/c Queues M/M/c Average time spent in system
λ = arrival rate μ = departure rate

56. M/M/c Example Problem l 200 m 250 c 2 r 0.8 P0 0.42857 Lq 0.15238 L Wq W

57. Multiple Processor Arrival Rate
200
250
300
350
400
450
475 m c 2 r
0.8 1
1.2
1.4
1.6
1.8
1.9 P0
0.25 Lq
0.675
1.3451 L
1.875
2.7451 Wq W

58. System Size vs Utilization

59. Adding Processors l 475 m 250 c 2 3 4 5 6 r 1.9 P0 0.02564 0.16618 Lq
0.0631 L
1.9631 Wq
0.0005
3.1E-05 W
0.0045

60. Time to Process vs Number of Servers

61. System Size Probabilitiesn Pn
Probability
Cumulative Probability P0 1 P1 2 P2 3 P3 4 P4 5 P5 6 P6 7 P7 8 P8 9 P9 10
P10 11
P11 12
P12 13
P13 14
P14 15
P15 16
P16 17
P17 18
P18 19
P19 20
P20

62. M/M/c Observations M/M/c is the general case
M/M/c behaves similar to M/M/1
Adding processors reduces queue length and time
However the System time can never be smaller than the processor time
An infinite number of servers will have system time = processor time

63. Classic Design Question
If my system is performing unsatisfactorally, do I add another processor or trade the current processor in for one that is twice as fast?

64. M/M/1 vs M/M/2 M/M/1 M/M/2 l 240 m 250 500 c 1 2 r 0.96 0.48 P0 0.2 Lq
23.04 L 24 Wq
0.096 W
0.1

65. Observations
Twice as fast is better because it reduces both the queuing time and the mean processing time