1. 1 Networks of queues Networks of queues reversibility, output theorem, tandem networks, partial balance, product-form distribution, blocking, insensitivity, BCMP networks, mean-value analysis, Norton's theorem Richard J. Boucherie Stochastic Operations Research department of Applied Mathematics University of Twente

2. 2 Networks of Queues Nelson, sec 10.1 Continuous time Markov chain Birth-death process Example: M/M/1 queue Birth-death process: equilibrium distribution Reversibility, stationarity Time reversed process

3. 3 Continuous time Markov chain stochastic process X=(X(t), t≥0) evolution random variable countable or finite state space S ={0,1,2,…} Markov property Markov chain: Stochastic process satisfying Markov property Transition probability time homogeneous Chapman-Kolmogorov equations

4. 4 Continuous time Markov chain Chapman-Kolmogorov equations transition rates or jump rates Kolmogorov forward equations: (REGULAR)

5. 5 Continuous time Markov chain Assume ergodic and regular global balance equations (equilibrium equations) π is invariant measure (stationary measure) solution that can be normalised is stationary distribution if stationary distribution exists, then it is unique and is limiting distribution

6. 6 Markov jump chain For Markov chain: holding time in state i is exponentially distributed; it is minimum of exponential times for each transition out of state i, say to state j with rate q(i,j) Minimum has exponential distribution with rate q(i) =-q(i,i) Probability that transition to state j occurs upon departure from state i is p(i,j)=q(i,j) / q(i) Markov jump chain : Markov chain with transition rates specified by q(i), and p(i,j) and is equivalent to original chain For Markov jump chain, we may generalise the holding time in the states to non-exponential times: this does not affect the equilibrium probability for being in state i

7. 7 Networks of queues Today (lecture 1): Continuous time Markov chain Birth-death process Example: M/M/1 queue Birth-death process: equilibrium distribution Reversibility, stationarity Time reversed process

8. 8 Birth-death process State space Markov chain, transition rates Kolmogorov forward equations Global balance equations

9. 9 Networks of queues Today (lecture 1): Continuous time Markov chain Birth-death process Example: M/M/1 queue Birth-death process: equilibrium distribution Reversibility, stationarity Time reversed process

10. 10 M/M/1 queue Poisson arrival process rate λ, single server, exponential service times, mean 1/μ State space S ={0,1,2,…} Markov chain? Assume initially empty: P ( X (0)=0)=1, Transition rates :

11. 11 M/M/1 queue Kolmogorov forward equations, j>0 Global balance equations, j>0

12. 12 M/M/1 queue λ λ λ j j+1 μ μ Equilibrium distribution: λ<μ Stationary measure; summable  equilibrium distribution Proof: Insert into global balance Detailed balance! Theorem: A distribution that satisfies detailed balance is a stationary distribution

13. 13 Networks of queues Today (lecture 1): Continuous time Markov chain Birth-death process Example: M/M/1 queue Birth-death process: equilibrium distribution Reversibility, stationarity Time reversed process

14. 14 Birth-death process State space Markov chain, transition rates Definition: Detailed balance equations Theorem: Assume that then is the equilibrium distrubution of the birth-death process X.

15. 15 Networks of queues Today (lecture 1): Continuous time Markov chain Birth-death process Example: M/M/1 queue Birth-death process: equilibrium distribution Reversibility, stationarity Time reversed process

16. 16 Reversibility; stationarity Stationary process: A stochastic process is stationary if for all t 1,…,t n,τ Theorem: If the initial distribution of a Markov chain is a stationary distribution, then a Markov chain is stationary Reversible process: A stochastic process is reversible if for all t 1,…,t n,τ

17. 17 Reversibility; stationarity Lemma: A reversible process is stationary. Theorem: A stationary Markov chain is reversible if and only if there exists a collection of positive numbers π(j), jεS, summing to unity that satisfy the detailed balance equations When there exists such a collection π(j), jεS, it is the equilibrium distribution. Proof

18. 18 Proof suppose process reversible Let rev.: then Conversely: suppose there exists π satisfying detailed balance, summing gives global balance.

19. 19 Proof (ctd.) Consider behaviour of X(t) for -T≤t ≤T starts at –T in j 1, stays random time h 1 before jumping to j 2, stays random time h 2, before jumping to j 3, and so on, until it arrives at j m, where it stays until T Probability density for this path is density with respect to h 1,…h m. Integrate over h 1,…h m such that h 1 +…+h m =2T. Detailed balance implies so that (*) is prob density for path starting –T in j m, stays random time h m before jumping to j m-1, and so on, until it arrives at j 1, stays until T. Thus, also invoking stationarity, (*)

20. 20 Kolmogorov’s criteria Theorem: A stationary Markov chain is reversible iff for each finite sequence of states. Furthermore, for each sequence for which the denominator is positive.

21. 21 Networks of queues Today (lecture 1): Continuous time Markov chain Birth-death process Example: M/M/1 queue Birth-death process: equilibrium distribution Reversibility, stationarity Time reversed process

22. 22 Time reversed process X(t) reversible Markov process  X(-t) also, but time homogeneity not inherited for non-stationary process Lemma: If X(t) is a time-homogeneous Markov process which is non stationary, then the reversed process X(τ-t) is a Markov process which is not even time-homogeneous. Proof. X(t) is a Markov process  X(τ-t) easy non-time-homogeneous: observe does not depend on t (time-hom) do depend on t at least for some t,j,k and so also the transition probabilities of the reversed process

23. 23 Time reversed process Theorem: If X(t) is a stationary Markov process with transition rates q(j,k), and equilibrium distribution π(j), jεS, then the reversed process X(τ-t) is a stationary Markov process with transition rates and the same equilibrium distribution. Theorem: Kelly’s lemma (Proposition 10.2) Let X(t) be a stationary Markov process with transition rates q(j,k). If we can find a collection of numbers q’(j,k) such that q’(j)=q(j), jεS, and a collection of positive numbers π(j), jεS, summing to unity, such that then q’(j,k) are the transition rates of the time-reversed process, and π(j), jεS, is the equilibrium distribution of both processes.

24. 24 Networks of queues Today (lecture 1): Continuous time Markov chain Birth-death process Example: M/M/1 queue Birth-death process: equilibrium distribution Reversibility, stationarity Time reversed process

25. 25 Summary / next: Detailed balance or reversibility and their consequences Birth-death process M/M/1 queue Kolmogorov’s criteria Next on AQT….. Section 10.2 – 10.3.5