1. Networks of queues
Networks of queues reversibility, output theorem, tandem networks, partial balance, product-form distribution, blocking, insensitivity, BCMP networks, mean-value analysis, Norton's theorem
Richard J. Boucherie
Stochastic Operations Research
department of Applied Mathematics
University of Twente

2. Networks of Queues Nelson, sec 10.1 Continuous time Markov chain
Birth-death process
Example: M/M/1 queue
Birth-death process: equilibrium distribution
Reversibility, stationarity
Time reversed process

3. Continuous time Markov chain
stochastic process X=(X(t), t≥0) evolution random variable
countable or finite state space S={0,1,2,…}
Markov property
Markov chain: Stochastic process satisfying Markov property
Transition probability time homogeneous
Chapman-Kolmogorov equations

4. Continuous time Markov chain
Chapman-Kolmogorov equations
transition rates or jump rates
Kolmogorov forward equations: (REGULAR)

5. Continuous time Markov chain
Assume ergodic and regular
global balance equations (equilibrium equations)
π is invariant measure (stationary measure)
solution that can be normalised is stationary distribution
if stationary distribution exists, then it is unique and is limiting distribution

6. Markov jump chain
For Markov chain: holding time in state i is exponentially distributed; it is minimum of exponential times for each transition out of state i, say to state j with rate q(i,j)
Minimum has exponential distribution with rate q(i) =-q(i,i)
Probability that transition to state j occurs upon departure from state i is p(i,j)=q(i,j) / q(i)
Markov jump chain : Markov chain with transition rates specified by q(i), and p(i,j) and is equivalent to original chain
For Markov jump chain, we may generalise the holding time in the states to non-exponential times: this does not affect the equilibrium probability for being in state i

7. Networks of queues Today (lecture 1):
Continuous time Markov chain
Birth-death process
Example: M/M/1 queue
Birth-death process: equilibrium distribution
Reversibility, stationarity
Time reversed process

8. Birth-death process State space Markov chain, transition rates
Kolmogorov forward equations
Global balance equations

9. Networks of queues Today (lecture 1):
Continuous time Markov chain
Birth-death process
Example: M/M/1 queue
Birth-death process: equilibrium distribution
Reversibility, stationarity
Time reversed process

10. M/M/1 queue
Poisson arrival process rate λ, single server, exponential service times, mean 1/μ
State space S={0,1,2,…}
Markov chain?
Assume initially empty: P(X(0)=0)=1,
Transition rates :

11. M/M/1 queue Kolmogorov forward equations, j>0
Global balance equations, j>0

12. M/M/1 queue Equilibrium distribution: λ<μ
λ λ λ
j j+1 μ μ
Equilibrium distribution: λ<μ
Stationary measure; summable  equilibrium distribution
Proof: Insert into global balance
Detailed balance!
Theorem: A distribution that satisfies detailed balance is a stationary distribution

13. Networks of queues Today (lecture 1):
Continuous time Markov chain
Birth-death process
Example: M/M/1 queue
Birth-death process: equilibrium distribution
Reversibility, stationarity
Time reversed process

14. Birth-death process State space Markov chain, transition rates
Definition: Detailed balance equations
Theorem: Assume that then is the equilibrium distrubution of the birth-death process X.

15. Networks of queues Today (lecture 1):
Continuous time Markov chain
Birth-death process
Example: M/M/1 queue
Birth-death process: equilibrium distribution
Reversibility, stationarity
Time reversed process

16. Reversibility; stationarity
Stationary process: A stochastic process is stationary if for all t1,…,tn,τ
Theorem: If the initial distribution of a Markov chain is a stationary distribution, then a Markov chain is stationary
Reversible process: A stochastic process is reversible if for all t1,…,tn,τ

17. Reversibility; stationarity
Lemma: A reversible process is stationary.
Theorem: A stationary Markov chain is reversible if and only if there exists a collection of positive numbers π(j), jεS, summing to unity that satisfy the detailed balance equations When there exists such a collection π(j), jεS, it is the equilibrium distribution.
Proof

18. Proof suppose process reversible
Let rev.: then
Conversely: suppose there exists π satisfying detailed balance, summing gives global balance.

19. Proof (ctd.) Consider behaviour of X(t) for -T≤t ≤T starts at –T in j1, stays random time h1 before jumping to j2, stays random time h2, before jumping to j3, and so on, until it arrives at jm, where it stays until T Probability density for this path is
density with respect to h1,…hm. Integrate over h1,…hm such that
h1+…+hm=2T. Detailed balance implies so that (*) is prob density for path starting –T in jm, stays random time hm before jumping to jm-1, and so on, until it arrives at j1, stays until T. Thus, also invoking stationarity,
(*)

20. Kolmogorov’s criteria
Theorem: A stationary Markov chain is reversible iff for each finite sequence of states. Furthermore,
for each sequence for which the denominator is positive.

21. Networks of queues Today (lecture 1):
Continuous time Markov chain
Birth-death process
Example: M/M/1 queue
Birth-death process: equilibrium distribution
Reversibility, stationarity
Time reversed process

22. Time reversed process
X(t) reversible Markov process  X(-t) also, but time homogeneity not inherited for non-stationary process
Lemma: If X(t) is a time-homogeneous Markov process which is non stationary, then the reversed process X(τ-t) is a Markov process which is not even time-homogeneous.
Proof. X(t) is a Markov process X(τ-t) easy non-time-homogeneous: observe does not depend on t (time-hom) do depend on t at least for some t,j,k and so also the transition probabilities of the reversed process

23. Time reversed process
Theorem: If X(t) is a stationary Markov process with transition rates q(j,k), and equilibrium distribution π(j), jεS, then the reversed process X(τ-t) is a stationary Markov process with transition rates and the same equilibrium distribution.
Theorem: Kelly’s lemma (Proposition 10.2) Let X(t) be a stationary Markov process with transition rates q(j,k). If we can find a collection of numbers q’(j,k) such that q’(j)=q(j), jεS, and a collection of positive numbers π(j), jεS, summing to unity, such that then q’(j,k) are the transition rates of the time-reversed process, and π(j), jεS, is the equilibrium distribution of both processes.

24. Networks of queues Today (lecture 1):
Continuous time Markov chain
Birth-death process
Example: M/M/1 queue
Birth-death process: equilibrium distribution
Reversibility, stationarity
Time reversed process

25. Summary / next:
Detailed balance or reversibility and their consequences Birth-death process M/M/1 queue Kolmogorov’s criteria
Next on AQT….. Section 10.2 –