CSCI1600: Embedded and Real Time Software Lecture 19: Queuing Theory Steven Reiss, Fall 2015

Queuing Theory History  Agner Krarup Erlang ( )  Analysis of telephone networks ( )  Why does this involve queues?

Analysis of Telelphone Networks  Issues  How many operators are needed  How many circuits are needed

Queuing Theory  A branch of probability theory studying queues  Applications  Computer systems  Communication networks  Call centers  Transportation  … Arriving Jobs Server Departing Jobs

Parameters: Arrivals  Arrival rate λ  How fast jobs are arriving  Regular or random  Random = Poisson process  With some distribution Arriving Jobs Server Departing Jobs

Parameters: service  Service demand S – amount of service the job needs  Measured in various terms (time, size, …)  Regular or random (with a given distribution)  Service capacity C – how fast server process jobs  Service rate μ = C/S  Service time = S/C = 1/μ Arriving Jobs Server Departing Jobs

Parameter: Service  Multiple symmetric servers  m parallel servers constitute a single serving station  Each has capacity C  Overall capacity depends on number of serviced jobs  C(N) = min(N,m)C Arriving Jobs Server 1 Departing Jobs Server m

Parameters: Queue  Scheduling discipline  First come, first served (FCFS, FIFO)  Priority queue (Shortest in, first out)  OS I/O scheduler  Preemptive vs. nonpreemptive servicing  Preemptive servicing can be interrupted by higher priority job  Linux real-time thread scheduling Arriving Jobs Server Departing Jobs

Statistical Basis for Queuing Theory  Arrival probability distribution  Service probability distribution  Determine various properties of the system  Average queue size  Maximum expected queue size  Average wait time  Maximum expected wait time

Poisson Arrival  A completely random arrival process  The number of jobs is very large  Jobs are independent of each other  Job arrivals are not simultaneous  λ: the average arrival rate  Examples  Phone calls to a switchboard  Radioactive decay  Customers going to checkout  Trains needing switches thrown

Poisson Process  Represented by Poisson distribution  Probablity of seeing i jobs during the time interval T  P[n(t) = i] = (λT) i /i! * e -λT PDF CDF

Poisson Process Properties  Memoryless  The number of arrivals in the time interval (T;T+Δt)  Does not depend on the number before T  Superposition  Combination of m independent Poisson processes with λ 1,λ 2,…λ m is a Poisson process with λ= λ 1 +λ 2 +…+λ m  Interval between job arrivals  Is distributed exponentially f x (x) = λe -λx

Exponential Service Distribution  PDF: fs(t) = μe -μt where μ is the service rate  CDF: Fs(t) = 1 - μe -μt  Mean = Std. Deviation = 1/μ = service time

Other Distributions  Erlang  The job visits an exponential server k times  Hyperexponential  k types of jobs, each has probability π i  Each type of job has its own service time  Each type of jobs passed to its own server  General  With known mean and standard deviation

A/S/n Notation  Proposed by D. Kendeall in 1953  Arrival process (A) – message arrival distribution  Server process (S) – servicing time distribution  Number of servers (n)  Distribution lables  M : exponential (memoryless, Poisson)  D : deterministic (constant arrival/processing times)  G : general  E k : k-stage Erlang  H k : k-stage Hyperexponential

Examples  M/M/1  One server for jobs that arrive and are serviced by Poisson process  System with many independent jobs sources can be approximated this way  M/D/n  Arrival process is Poisson, service is deterministic  There are n servers  G/G/n  General system with arbitrary arrival and service times

Goals of Queuing Theory  Predict the performance of the system in steady state  Not startup or shut down  Assumptions  The population of jobs is infinite  The size of the queue is infinite (can be bounded)  Specific distributions for A and S known

Performance Metrics  W: waiting time (queuing delay)  Time between receipt of a job and start of processing  Depends on the load on the server  S: service time (processing delay)  The actual processing time of the job  Depends on the serving capacity  T: response time  T = W + S  Throughput  Average number of jobs per time unit

Little’s Law  Number of jobs in the system N(t)  N = λT  λ : the mean arrival rate  T : the mean service time  Queue size Q  Q = max(0,N – m)  Waiting time in the Queue W  Q = λW

Little’s Law Example  Clothing store  Customers arrive 10/hour and stay 30 minutes  λ = 10, T = 0.5  Average number of customers in the store N = 10*0.5 = 5  Register line subsystem in the store  Average register queue is 2 people  The wait time W = Q/λ = 2/10 = 0.2 hours

Other Metrics  Traffic intensity p  Single server: p = arrival rate * service time = λ/μ  m servers: p = λ/(mμ)  p is dimensionless, measured in Erlangs  If p > 1, then more jobs arrive than the system can handle  p < 1 needed to prevent overload  Utilization factor  U = min(p,1)

Analytic Solutions  Some queuing models can be solved analytically  M/M/1 for example  Other variants

M/M/1 Performance

M/M/1 Example

How to Fix Overload  How can we reduce job delays  Obvious:  Shorten μ (speed up the code, profile, optimize)  Lengthen λ (decrease the load, make less bursty)  Move to M/M/m  How many servers do you want  This can be solved as well

Problems  Classical queuing theory is too restrictive  Infinite number of jobs  Only certain distributions are handled  Job routine is not allowed  Some limitations can be alleviated  But calculations and the math involved are complex  Provides insufficient information

Solving Queuing Problems  Use a computer representation of the system  Instead of a mathematical model  Use simulation to learn the properties  This is what is generally used  MATLAB SimEvents  OMNET  Can construct arbitrary models and run them

Homework  For next week (10/26)  Present you project to the class  Plans, progress to date  Project model  Tasks and model(s) for each task  Hand this in  Can be part of presentation