IEG4140 Teletraffic Engineering

IEG4140 Teletraffic Engineering
Professor: S.-Y. Robert Li Rm. HSH734 (inside 727), x8369, Tutor: Diana Q. Wang Rm. HSH729, Visit course website: frequently to catch spontaneous announcements. Textbook: Lecture notes on course website, under constant revision. Reference Book: Chapters 4~8 of S. Ross, “Probability Models,” 8th ed., Academic Press

Assessment scheme: The grading emphasizes on logical reasoning rather than numerical calculation:
10% homework 30% mid-term exam 60% final exam Bonus for actively asking questions in the classroom Prof. Bob Li

Lecture notes: Chap 1. Motivating puzzles, markov chain, stopping time, and martingale Chap 2. Poisson process Chap 3. Continuous-time markov chain Chap 4. Renewal process Chap 5. Queueing theory Prof. Bob Li

Schedule: Time Content Week 1 (Sep 8)
Some puzzles in stochastic processes & markov Chain I Week 2 (Sep 15) Markov Chain II Week 3 (Sep 22) Time Reversed markov Chains Week 4 (Sep 29) Stopping time, Wald’s equation & Martingale Week 5 (Oct 6) Exponential and Poisson Distribution I Week 6 (Oct 13) Exponential and Poisson Distribution II Week 7 (Oct 20) The Poisson Process I Week 8 (Oct 27) Midterm exam (tentative date) Week 9 (Nov 3) The Poisson Process II Week 10 (Nov 10) Continuous-time markov Chain I Week 11 (Nov 17) Continuous-time markov Chain II Week 12 (Nov 24) Queuing Theory I Week 13 (Dec 1) Queuing Theory II Prof. Bob Li

Lecture 1 Sep. 8, 2011 Prof. Bob Li

Chapter 1. Introductionmotivating puzzles, markov chain, stopping time, and martingale
1.1. Some puzzles in stochastic processes 1.2. Markov chain 1.3. Time reversed markov chain 1.4. Markov Decision Process 1.5. Stopping Time 1.6 Martingales Prof. Bob Li

1. 1 Some puzzles in stochastic processes 1. 1. 1
1.1 Some puzzles in stochastic processes Pattern occurrence in repeated coin toss P{Head} = P{Tail} = ½. In a very long binary sequence, all the 64 length-6 patterns appear equally frequently. HTHTTTTHTHHTHTTTHTTTTHTHHTHTTTHTHTHTTHTTHTHTTTHTHHTHTTHTHHTHHTHHTHTTHTHHTHHTHHTHTTTHTHHTTHHTTTHTTTTTHTHHTHHTHHTHTTHTHHTHTTTTHTHHTHTTTHTHHTHHTHHTTTHHTHHTHTHTTHTHTHHTHTHTTHTHHHHTHTHTTHTTHTTTHTHTHTHTHHTTTTHHTHTTHTHTHTHTTHHTHHTHTHTHTHHTHTTTHTTTHHTHTHTHHTTHHTHTTHTTHTTHTHHTHHTHHTHTTHTTHTHTHTHTTHTHTHTTHTHTHHTTHTTHTHTHTTHTHTHHHTHHTHTHTTHTTHTHHTHTTHTHHTHTHHTHTHTTHTTHTHTHTHTTHTHTHTHTTHTHTTTTHTHHTTHHHTTHHTTTHHHHTHHTHTHHHHHHTTHTHTHTHHTHHTHTTHHTHTHTTHHTHTHTTTHTHTHHTHTHHTTHTHHTTHHTTHHHTTHTTHTTHTTHHTHHTHTTHTTHTTHTHTHHTHTHTTHTTHTHTTHTTHTTHTHHTHTTHTTTHTHTTHTHHTTH … So, the average waiting time for every length-6 pattern should be 64. Right? Prof. Bob Li

William Feller’s biblical textbook
The book has first raised these questions: Average waiting time for HHHHHH = 64? Average waiting time for HHTTHH = 64? Feller did not live to see his 2-volume book. Prof. Bob Li

William Feller’s biblical textbook
Through long derivation by markov chain, Average waiting time for HHHHHH = 64? Average waiting time for HHTTHH = 64? William Feller found these numbers counter-intuitive. 126 70 Feller did not live to see his 2-volume book. Prof. Bob Li

Paradoxes in pattern occurrence
HTHTTTTHTHHTHTTTHTTTTHTHHTHTTTHTHTHTTHTTHTHTTTHTHHTHTTHTHHTHHTHHTHTTHTHHTHHTHHTHTTTHTHHTTHHTTTHTTTTTHTHHTHHTHHTHTTHTHHTHTTTTHTHHTHTTTHTHHTHHTHHTTTHHTHHTHTHTTHTHTHHTHTHTTHTHHHHTHTHTTHTTHTTTHTHTHTHTHHTTTTHHTHTTHTHTHTHTTHHTHHTHTHTHTHHTHTTTHTTTHHTHTHTHHTTHHTHTTHTTHTTHTHHTHHTHHTHTTHTTHTHTHTHTTHTHTHTTHTHTHHTTHTTHTHTHTTHTHTHHHTHHTHTHTTHTTHTHHTHTTHTHHTHTHHTHTHTTHTTHTHTHTHTTHTHTHTHTTHTHTTTTHTHHTTHHHTTHHTTTHHHHTHHTHTHHHHHHTTHTHTHTHHTHHTHTTHHTHTHTTHHTHTHTTTHTHTHHTHTHHTTHTHHTTHHTTHHHTTHTTHTTHTTHHTHHTHTTHTTHTTHTHTHHTHTHTTHTTHTHTTHTTHTTHTHHTHTTHTTTHTHTTHTHHTTH … In the long run, the pattern HHTTHH occurs equally frequently as any other length-6 patterns and hence it occurs after every 64 coin tosses on the average. In more precise terms, the renewal time averages 64 (and  4), which is measured from one appearance of the pattern till the next. Prof. Bob Li

Intuition vs. misconception
For a length-6 binary pattern: The renewal time, measured from one appearance to the next, averages 64. The waiting time, measured from the very beginning of the coin-toss process, can be longer. Quick intuition defies scientific truth when it mixes up the two concepts. Let’s see how far can quick intuition be away from scientific truth? Prof. Bob Li

Race between 2 patterns On fair coin-toss,
Average waiting for HTHH = 18 // A slightly faster pattern Average waiting for THTH = 20 // A slightly slower pattern In a race between HTHH vs. THTH, the odds are nothing like 10 : 9, but rather … Prof. Bob Li

Race between 2 patterns On fair coin-toss,
Average waiting for HTHH = 18 // A slightly faster pattern Average waiting for THTH = 20 // A slightly slower pattern In a race between HTHH vs. THTH, the odds are the landslide 5 : 9. nothing like 10 : 9, but rather … Prof. Bob Li

The Sting The fairy-tale race On fair coin-toss,
Average waiting for HTHH = 18 // A slightly faster pattern Average waiting for THTH = 20 // A slightly slower pattern In a race between HTHH vs. THTH, the odds are the landslide 5 : 9 in favor of THTH. Hare Tortoise Scientific truth and fairy tales happily agree with each other, while ordinary intuition is left out.

It’s a bit like 田忌賽馬 Tortoise often wins by a neck, while a Hare’s win is at least a full length. Homework. Compute P(HH occurs before TTT) with a biased coin where P(Head) = 0.4 . Later we’ll deal with this topic again by martingales.

1.1.2. An imaginary casino Pay $100 to enter the game:
Repeatedly toss a fair coin until a Head shows up. Receive $2n back if Head occurs on the nth toss. E[Return] = = = $ // E[N] can be , even when P(N < ) = 1. Q. Is this $ vs. $100 a huge advantage to the gambler in practice?

Symmetric random walk Abel & Billy gamble $1 on each toss of a fair coin. This is a zero-sum game. The net winning of Abel is a symmetric random walk. P{Abel nets +$1 sooner or later, i.e., time till +$1 is finite} =1 ? The moral. Despite the 100% probability for the occurrence of this event in finite time, that finite time is unbounded and, in fact, has an mean value = . On the average, it takes  tosses to achieve the net gain of $1. The average net gain per toss is $1/ = $0, as it should be in fair gamble.

Symmetric random walk (cont’d)
Abel & Billy gamble $1 on each toss of a fair coin. This is a zero-sum game. The net winning of Abel is a symmetric random walk. P{Time till Abel nets +$1 is finite} = 1 ? P{Time till Abel nets +$1000 is finite} = 1 ? P{Time till Billy nets +$1000 is finite} = 1 ? Answer: Yes to all, even though it is a zero-sum game.

1.1.4. 2-dimensional random walk
The street map of Manhattan is an infinite checker board. A drunkard starts a symmetric 2-dimensional random walk at the street corner outside the pub. Q. P{Return to the origin within finite time} = 1? 1/4 Answer: Yes. Prof. Bob Li

1.1.5. 3- or higher-dim random walk
Consider an infinite 3-dimensional checker board. At every cross point, the six directions are equally likely: east, west, north, south, up, and down Q1. P{Return to the origin within finite time} =1? Answer: No. Q2. How about 4-dim random walk? Prof. Bob Li

Progressive gambler On casino roulette, a gambler keeps betting on "odd", where P(Odd) = P({1, 3, ... , 35}) = 18/38 P(Not odd) = P({0, 00, 2, 4, ... , 36}) = 20/38 A gambler starts with a $1 bet. Whenever he loses, he doubles the bet on the next spin. This process continues until he wins, at which time, he nets +$1 regardless of how long the process takes. Q. Is this a sure way to win $1 from the casino? Answer: It is a sure way to enrich the casino. Prof. Bob Li

A board game Each move advances the token by a number of steps determined by rolling a die. P{Square #1 will be landed on} = 1/6 P{Square #2 will be landed on} = 1/6 + 1/36 = 7/36 Q. Which square has the highest probability to be ever landed on?

A board game (cont’d) Let pn = probability of ever landing on the square n. By conditioning on the first roll, pn= 1k6 P(1st roll = k)  P(ever landing on n | 1st roll = k) = 1k6 P(ever landing on n | 1st roll = k) /6 = (pn-1 + … + pn-6) /6 That is, every probability pn, n  1, is the average of its six immediate predecessors. The recursive formula is a 6th-order difference equation. There are 6 boundary conditions: p0 = 1, p-1 = p-2 = p-3 = p-4 = p-5 = 0 In this course, we shall solve difference equations quite often.

A board game (cont’d) Values of pn are tabulated below.
The largest among p1 to p6 is p6. Because every probability mass pn is the average of its six immediate predecessors, it is an easy induction to find pn < p6 for all n > 6. n pn -5 1 1/6 = .1667 7 = .2536 13 = .2790 -4 2 7/62 = .1944 8 = .2681 14 = .2832 -3 3 72/63 =.2269 9 = .2804 15 = .2857 -2 4 73/64 = .2647 10 = .2891 16 = .2866 -1 5 74/65 = .3088 11 = .2932 … 6 75/66 = .3602 12 = .2906   2/7= .2857

A board game (cont’d) Homework. Show that p13  pn  p11 for all n > 13. Homework. If we roll two dice instead of just one, which square has the highest probability to be landed on? Hint: Find the largest among p1 to p12. Remark. This board game can be modeled as a markov chain in the obvious way. Conditioning on the first roll is the same as a markov transition, which is also the same as multiplication by the transition matrix. Prof. Bob Li

1.2 Markov chains Definition. A sequence of discrete r.v. X0, X1, X2, ... is called a markov chain if there exists a matrix such that // P(Xt+1 = j | Xt = i) is time-invariant. // Memoryless: Only the newest knowledge Xt counts. Prof. Bob Li

Memoryless and time-invariant
Definition. A sequence of discrete r.v. X0, X1, X2, ... is called a markov chain if there exists a matrix such that // P(Xt+1 = j | Xt = i) is time-invariant. // Memoryless: Only the newest knowledge Xt counts. Remarks. Any possible value of any Xt is called a state of the markov chain. When there are k ( ) states in the markov chain, the matrix  is square (k by k) and is called the transition matrix of the markov chain. Pij is called the transition probability from state i to state j.

Example of 3-state weather model
A simple 3-state model of weather assumes that the weather on any day depends only upon the weather of the preceding day. This model can be described as a 3-state markov chain with the transition matrix The transition graph (or transition diagram) is: To sunny rainy cloudy From Prof. Bob Li

Every row sum = 1 in transition matrix
The transition equation Pij = P(Xt+1= j | Xt= i) says that the ith row of the transition matrix is the conditional distribution of Xt+1 given that Xt = i. Hence the row sum must be 1. In contrast, column sum is in general not 1. To sunny rainy cloudy From Row sum =1 Row sum =1 Row sum =1 Column sum  1 Prof. Bob Li

Free 1-dimensional Random Walk
At each game, the gambler wins $1 with the probability p and loses $1 with the probability q = 1p. Let Xt be the total net winning after t games. Thus the states are integers and Modeling by a markov chain, the transition matrix and transition graph are: j = i i

Random Walk with an absorbing state
In the same gamble as before, the gambler starts with $2. He has to stop gambling if the net winning reaches $2. In other words, P-2,-2 = 1 and P-2,j = 0 for all j  2. The transition matrix and diagram for the markov chain are: i j 2  - é ë ê ù û ú 2 1 K q p Prof. Bob Li

Independence of Partial Memory
P(X3= j | X2= i, X1= i1, X0= i0) = Pij To calculate the probabilities for X3, if we know X2, then we can throw away the knowledge about both X1 and X0. Example. P(X3= j | X2= i, X0= i0) = Pij // Only the newest knowledge counts. Proof. Conditioning on X1, P(X3= j | X2= i, X0= i0) Problem. Prove that P(X3= j | X2= i, X1= i1) = Pij by conditioning on X0. // Conditioning on X1 = Pij

Independence of Partial Memory (cont’d)
Example. P(X3= j | X1= i, X0= i0) = ? Answer. Conditioning on X2, P(X3= j | X1= i, X0= i0) // By conditioning on X2 = The ijth entry in 2 = P(X3= j | X1= i) // Special case of Chapman-Komogorov equation below Conclusion: Only the newest knowledge counts. Prof. Bob Li

Transition Equation in Matrix Form
Conditioning on Xt, we have where the summation is over all states i. In the matrix form, this becomes ,where the row vector denotes the distribution of Xt // Multiplication by  = a transition. Prof. Bob Li

Transition Equation in Matrix Form
Whether example. The notation of the row vector in this example is simply Vt = The transition equation in the matrix form becomes Prof. Bob Li

Chapman-Komogorov Equation
Now we iterate the transition equation starting with Vt. Vt+1 = Vt // Multiplication with  represents a transition. Vt+2 = Vt+1 = (Vt)  = Vt2 By induction on n, we arrive at the following Chapman-Komogorov Equation. The matrix n gives the n-step transitions. That is, Vt+n = Vtn. Note. By conditioning on Xt , Thus, P(Xt+n =j | Xt = i) = the ijth entry in the matrix n // For all t

Prisoner in Dark Cell Hole A leads to freedom through a tunnel of a 3-hour journey. Holes B and C are at the two ends of a tunnel of a 2-hour journey. Whenever the prisoner returns to the dark cell, he would immediately enter one of the three holes by a random choice. Q. When can the prisoner get out? Q: What should be the "states" in the model of a markov chain? Prof. Bob Li

Prisoner in Dark Cell (cont’d)
Label the “hour-points" as F, R, & T. Use these points, together with 'cell' and 'freedom' as states. Transition matrix Transition graph

Prisoner in Dark Cell (cont’d)
As before let Vt denote the distribution of Xt. Then, V0 = ( ) // Initially the prisoner is in the cell. V1 = V0 = (2/ / ) V2 = V1 = (0 2/ /3 0) = V02 V3 = V2 = (4/ / /3) = V0 // 1/3 = P(Free after 3 days) V4 = V3 = (0 4/ /9 1/3) = V04 V5 = V4 = (8/ / /9) = V0 // 5/9 = P(Free after 5 days) V6 = V5 = (0 8/ /27 5/9) = V06 ... V = V  = ( ) = V0  // 1 = P(Free eventually) Note that V0 means the second row in the matrix  = In fact, even with any other V0 (i.e., any distribution of the initial state), we would still have V = V = ( ) = V0 Thus every row in  is ( ).

Slotted Aloha multi-access protocol
Node A Node B Node C In every timeslot, transmission is successful when there is a unique node actively transmitting. An transmission attempt can be either a new packet or a retransmission. Every backlogged node in every timeslot reattempts transmission with a probability p. The time till reattempt is Geometric1(p): P(time = t) = (1p)t 1p for all t  1 Note. Geometric0(p): P(time = t) = (1p)t p for all t  0

Markov model of Slotted Aloha
Markov model. Using the number of backlogged nodes as the state of the system, a transition from state k is always to some state j ≥ k−1. 1 2 3 4 5 Assumption for the convenience of analysis. The probability ai that i new arriving packets intend transmission in a timeslot is fixed and is independent of the state. Every backlogged node in every timeslot reattempts transmission with an independent probability p.

Markov model. Using the number of backlogged nodes as the state of the system, a transition from state k is always to some state j ≥ k−1. 1 2 3 4 5 Assumption for the convenience of analysis. The probability ai that i new arriving packets intend transmission in a timeslot is fixed and is independent of the state. The probability bi that i of the backlogged attempt retransmission depends on the state k binomially:

Markov model. Using the number of backlogged nodes as the state of the system, a transition from state k is always to some state j ≥ k−1. The transition probabilities from state k are: Pk,k1 = a0b // k > 0; 1 reattempt; no new Pk,k = a0(1b1) //  1 reattempt; no new +a1b // no reattempt; 1 new Pk,k+1 = a1(1b0) // k > 0; ≥ 1 reattempt; 1 new For i  2, Pk,k+i = ai //  2 new Prof. Bob Li

1.4 Limiting probabilities of markov chain
Definition. In the markov chain X0, X1, X2, ..., let the row vector Vt represent the distribution of Xt. If there exists a row vector V such that regardless of the initial distribution V0, then the distribution V is called the limiting distribution of the markov chain. Remarks. The limiting distribution is also called the “stationary state” by its interpretation as a “random state.” = V0  for all V0. Taking V0 = ( ), we find V equal to 1st row in the matrix . Similarly, taking V0 = ( ), we find V equal to 2nd row in . In the same manner, V is equal to every row in .

Example of limiting probability
Example. For the prisoner in the dark cell, Prof. Bob Li

Ergodic markov chain Definition. A finite-state markov chain is ergodic if, for some t  1, all entries in the matrix t are nonzero. // For some particular t, the markov chain can go // from anywhere to anywhere in exactly t steps. Theorem. If a finite-state markov chain is ergodic, then the limiting distribution exists. // Ergodicity is a sufficient condition but, we shall use an // example to show that it is not a necessary condition. Prof. Bob Li

Example of ergodic markov chain
Example. A salesman travels between HK and Macau. HK Macau All entries in 2 are positive. Hence this markov chain is ergodic. Prof. Bob Li

Limiting probabilities = eigenvector
Since V = V0( ) = V0( ) = V0( ) = V, the limiting distribution is a (row) eigenvector of the transition matrix  with the eigenvalue 1. V is a probability distribution (sum of entries = 1). The normalization by 2) is crucial since any scalar multiple of V is also an eigenvector with the eigenvalue 1. A row vector with both properties is called the long-run distribution. In the ergodic case, V is the unique long-run distribution. Prof. Bob Li

Example of a 2-state markov chain
HK Macau All entries in 2 are positive. Hence this markov chain is ergodic. Since the limiting distribution V is a row eigenvector of  with eigenvalue 1, V  = V or V (  I) = (0 0 … 0) The existence of an eigenvector renders (  I) a singular matrix. Write V = (x y). Thus, x/2 + y = x x/ = y are linearly dependent equations. The span the full rank minus 1, because the eigenspace is 1-dimensional. Together with the normalization equation x + y = 1, We can solve x = 2/3 and y = 1/3.

Limiting distribution of a 3-state chain
= é ë ê ù û ú Õ 1 9 05 5 45 . 2 05 54 405 03 36 63 005 14 855 = é ë ê ù û ú Õ . Given , all entries in are positive. Hence the markov chain is ergodic. Write the limiting distribution as V = (x y z). We want to calculate x, y, and z. Since V is a distribution, x y z = 1, On the other hand, V has to be a row eigenvector of  with eigenvalue 1. Thus, we have the linearly dependent equations: 0.1x y = x 0.9x y + 0.1z = y 0.45y + 0.9z = z They are worth only two equations. Together with the normalization equation, we get V = ( ).

Color-Blindness Gene Color-blindness is X-genetic. There are two kinds of X genes: X0 = color blind X1 = normal Thus there are three types of female: X0X0 = color blind X0X1 = normal // X1 gene dominates X0 X1X1 = normal and two types of male: X0Y = color blind X1Y = normal Question. In a “stationary” society, a survey shows that approximately 10% of the male population is color blind. However, the survey also finds too few color blind females to make a percentage estimate. What can we do?

Color-Blindness Gene (cont’d)
Solution. From the percentage of color blind males, we assert the 1 : 9 ratio between the X0 gene and the X1 gene. Then, model the problem by the mother-to-daughter transition matrix : Daughter X0X0 X0X1 X1X1 X0X0 Mother X0X1 X1X1 From the nature of the problem, we know the markov chain is ergodic. The preceding calculation also proves this and, in fact, yields the limiting distribution V = ( ). Thus, 1% of females are color blind, while another 18% are carriers of the color blindness gene. Prof. Bob Li

Convolution decoding Digital communications often stipulate inspection of the pattern of the last few, say, 3 received bits. Assume each bit is independently random with P(bit = 1) = p. All 3-step transition probabilities are positive. Hence the markov chain is ergodic. The components of V are calculated as: These are not surprising at all because of the independence assumption. Prof. Bob Li

Example of a 2-state markov chain
( ) ú û ù ê ë é - = P 1 2 n ú û ù ê ë é = P 1 lim n The markov chain is not ergodic. Nevertheless, the limiting distribution V = (0 1) exists. Prof. Bob Li

Example of Oscillation
State 0 State 1 There exists zero entries in t for all t. Hence the markov chain is not ergodic, but rather periodic. In fact, there is no limiting distribution. Let V0 = (1 0). Then Vt alternates between (1 0) and (0 1). Nevertheless, there exists the eigenvector ( ) of the transition matrix  with the eigenvalue 1. Prof. Bob Li

Ergodicity, limiting distributions, eigenvector
Summary of theorems. For finite-state markov chains, Ergodicity // All entries in the matrix t are nonzero for some t  1. The limiting distribution V exists. // regardless of the initial distribution V0. Eigenvalue 1 of the transition matrix with 1-dim eigenspace // Rank of the matrix I is full minus 1 (dim of eigenspace = 1). // Long-run distr. = unique normalized eigenvector with eigenvalue 1 Eigenvalue 1 of the transition matrix // The matrix I is a singular.  det(I) = 0 Q. When there is no limiting distribution, there may or may not be the eigenvalue 1. What happens then? See the next example. Prof. Bob Li

A gambler wants to go home.
A gambler in Macau needs $3 to return to HK but has only $2. He gambles $1 at a time, and wins with probability p.  = The matrix I has the rank 2 (< the full rank 4). There are two linearly independent eigenvectors of eigenvalue 1: ( ) and ( ). However, there is no limiting distribution. The probability of ending in the state $3 instead of $0 depends on the initial distribution V0. For the gambler, V0 = ( ). 1 2 3 p 1p Prof. Bob Li

Classification of states in a markov chain
Definition. For the markov chain X0, X1, X2, ... The probability of recurrence at a state i is fi = P(Xt = i for some t > 0 | X0 = i)  1 The state i is said to be recurrent when fi = 1 and transient when fi < 1. // For sure, the state will be revisited sooner or later. The multiplicity of visiting state i, denoted by Mi ( ), is the number of indices t > 0 such that Xt = i. Theorem. The conditional r.v. (Mi | X0 = i) is: deterministically equal to  when fi = 1 Geometric0(1fi) distributed when fi < // That is, P(Mi = n | X0 = i) = fin(1fi) Intuitive proof. Think of every revisit to state i as either Head or Tail depending whether it is the final visit. Thus P(Head) = 1 fi & P(Tail) = fi .

Classification of states (cont’d)
Computational Proof. Assuming that fi < 1, we shall prove by induction on n that P(Mi = n | X0 = i) = (1fi) fin P(Mi = 0 | X0 = i) = P(Xti for all t>0 | X0=i) = 1fi P(Mi= 1 | X0 = i) = t>0 {P(Mi=1 | X0 = i and 1st recurrence at time t)  P(1st recurrence at t | X0 = i)} = t>0 {P(no recurrence after time t | Xt = i)  P(1st recurrence at time t | X0 = i)} = t>0 {P(Mi=0 | X0 = i)  P(1st recurrence at time t | X0 = i)} // Time-invariant = (1fi) t>0 P(1st recurrence at time t | X0 = i)} = (1fi) fi P(Mi= n | X0 = i) = t>0 {P(Mi = n | X0 = i and 1st recurrence at time t)  P(1st recurrence at t | X0 = i)} = t>0 { P(n1 recurrence after time t | Xt = i)  P(1st recurrence at time t | X0 = i)} = t>0 {P(Mi = n1 | X0 = i)  P(1st recurrence at time t | X0 = i)} // Induction on n = (1fi) fin1 t>0 P(1st recurrence at time t | X0 = i)} = (1fi) fin // Geometric0(1fi) distribution

Corollary. E[Mi | X0= i] = 1 / (1fi) for fi  1. Corollary. E[Mi | X0= i] =  if and only if fi = 1. Proposition. E[Mi | X0 = i] = entry in the matrix t) Proof. Given i, let Yt be the characteristic r.v. of the event Xt = i. Thus, and hence E[Mi | X0 = i] = // Mean of sum is sum of means. = entry in the matrix t) // Chapman-Komogorov

Summary. fi = 1   entry in the matrix t) =  // Chapman-Komogorov // The final condition can be used to verify the recurrence of 1- // or 2-dim symmetric random walk but not of the 3-dim one. Corollary. In a finite-state markov chain, there is at least one recurrent state. In an infinite-state markov chain however, it is possible that all states are transient. We shall see that this is the case for the 1-dim asymmetric free random walk. Prof. Bob Li

Examples Example. In the 3-state weather model, all three states are recurrent. Example. In the markov chain for the prisoner in the dark cell, only the “freedom” state is recurrent. Prof. Bob Li

1-dim free random walk At each game, the gambler wins $1 with the probability p and loses $1 with the probability q = 1p. Let Xt be the total net winning after t games. Theorem. In an asymmetric 1-dimensional free random walk, every state is transient. In the symmetric 1-dimensional free random walk, every state is recurrent and hence the process almost surely will return to the initial state within finite time // “Almost surely” means “with probability 1.” Proof. Next slide. Prof. Bob Li

1-dim free random walk Proof. 0 for odd t when t = 2n
// To go from i to i in exactly 2n steps, there must // be exactly n steps to the right and n to the left. The state i is transient or recurrent hinges on the convergence the series = // t = 2n =  // Stirling's Formula: as n If p  1/2, then 4pq < 1 and the series converges. If p = 1/2, then 4pq = 1 and the series diverges. // By the “integral test”

1-dim symmetric random walk
Problem. Show that the symmetric random walk will almost surely reach the state to the right of the initial state. Solution = P[will ever return to 0 | X0 = 0] = (1/2) P[will ever return to 0 | X0 = 0 and X1 = 1] + (1/2) P[will ever return to 0 | X0 = 0 and X1 = 1] // Conditioning on X1 under the a priori of X0 = 0 = (1/2) P[will ever return to 0 | X1 = 1] + (1/2) P[will ever return to 0 | X1 = 1] // Memoryless property: only the newest information counts = P[will ever reach state 0 | X1 = 1] // by symmetry Prof. Bob Li

Positive recurrent states
Definition. For the markov chain X0, X1, X2, ... The waiting time Ti for state i is the smallest index t > 0 such that Xt = i. When there is no such an index t, define Ti = . The state i is positive recurrent if E[Ti | X0 = i] < . Theorem. Positive recurrent  recurrent Proof E[Ti | X0 = i] <   P(Ti =  | X0 = i) = 0  P(No recurrence | X0 = i) = 0  State i is recurrent. Prof. Bob Li

State classification Positive recurrent Recurrent Immortal 夫
Amah Rock (望夫石), Shatin, HK CUHK Recurrent Transient Positive recurrent Non-positive recurrent e.g., symmetric 1-dim random walk as we shall prove later Immortal 夫 Prof. Bob Li

Positive recurrent states
(cont’d) Positive recurrent states Theorem. Recurrent states in a finite-state markov chain are all positive recurrent. Proof. Omitted. Remarks. Fact. 2-dim symmetric free random walk is also recurrent. It cannot be positive recurrent. Why? Fact. 3-dim symmetric free random walk turns out to be transient. 4-dim symmetric free random walk is also transient. Why? Prof. Bob Li

Equivalent classes of states
Definition. State j is said to be accessible from state i if, starting from state i, the probability is nonzero for the markov chain to ever enter state j, that is, the ijth entry in the matrix t is positive for some t ≥ 0. If two states i and j are accessible to each other, we say that they communicate. // Mathematically, state communication is an “equivalence relation,” // that is: reflexive, symmetric, and transitive. Example. A 3-state markov chain with only one equivalent class of states. Prof. Bob Li

Equivalent classes of states (cont’d)
Example. A markov chain with four states in three equivalent classes: {0, 1}, {2} and {3}. Only state 2 is transient. Prof. Bob Li

Theorem. Recurrence/transience is a class property. // Thus it makes sense to mention a recurrent class or a transient class. Proof. Let states i and j belongs to an equivalent class. Write for the ijth entry in k. Thus and for some k and m. Suppose that i is recurrent, i.e We shall prove that so is j. Prof. Bob Li

Example. A markov chain with five states in three equivalent classes: {0, 1} is recurrent {2, 3} is recurrent {4} is transient Prof. Bob Li

skipEquivalent classes of states (cont’d)
Definition. When there is only one equivalence class, the markov chain is said to be irreducible. Theorem. All states of a finite-state irreducible markov chain are recurrent (and hence positive recurrent by an aforementioned theorem). Proof. Not all states can be transient in a finite-state markov chain. Problem. Prove that a transient state cannot be accessed from a recurrent state. Prof. Bob Li

SkipCalculation of expected time & multiplicity
Example of Gambler’s Ruin A gambler in Macau needs $7 to return to HK but has only $3. He starts to wage $1 at a time and wins with probability p = 0.4. The transition matrix of his fortune is as right One way to calculate fij = P(Xt = j for some t > 0 | X0 = i) is by first considering the following quantity si,j = E[number of visits to j | start at i] and by conditioning on whether the state j is ever entered (See p.197 of [Ross, 6th ed.] for detail.) Below, we calculate si,j for all i = 1 to 6.

skipExample of Gambler’s Ruin (cont’d)
Conditioning on the outcome of the initial play, si,j = i,j // The Knonecker delta i,j = 1 when i = j; else = 0 + p E[number of visits to j | start at i; initial play wins] + q E[number of visits to j | start at i; initial play loses] = i,j + p si+1,j + q si1,j Denote by S the 66 matrix (sij)1i6,1j6. Then, S = I + TS where I is the 66 identity matrix and T denotes the matrix within blue circumscription. Thus, I = S  TS = (I  T)S S = (I  T)1 As p = 0.4 and q = 0.6, we can explicitly calculate this inverse matrix and find, for instance, s3,2 = and s3,5 = .9228 Prof. Bob Li

Calculation of expected time till freedom
The transition graph The only recurrent state, i.e., the terminal state

Method 1: Given the initial state = cell, let the frequency of entering the internal tunnel be MT, which is Geometric0(1/3) distributed. Thus, E[Waiting time for freedom | initial state = cell] = E[2MT+3] = 2EMT+3 = 2*2+3 = 7 Method 2: Calculate from markov chain. For every state S, define the conditional r.v. TS = (Waiting time for freedom | initial state = S) and adopt the abbreviation eS = ETS. We have the following matrix equation. This matrix equation means conditioning on the outcome of the 1st step.

In other words, we have the following system of linear equations: From these, we can solve for eS for all states S. // Even though we only wanted to compute ecell, the // markov-chain computation gives eS for all states S. It turns out that eR = 1, eF = 2, ecell = 7, eT = 8 // Mean of sum is sum of means. Prof. Bob Li

Example of pattern occurrence in coin tossing
Q. In fair-coin tossing, the expected waiting time for the pattern HTHT to occur consecutively is not 16. A solid arrow means HEAD and a dotted means TAIL. Probability of every arrow is 0.5. For every state S, define the conditional r.v. TS = (waiting time for HTHT | initial state = S) and write eS = ETS. We have the following equation. The terminal state

Example of pattern occurrence in coin tossing
which implies // Mean of sum is sum of means. Solving the five simultaneous equations, it turns out that enull = 20. Alternative calculation of this quantity by martingale is instantly. See: S.-Y. R. Li, “A martingale approach to the study of occurrence of sequence patterns in repeated experiments,” Annals of Probability, v.8, pp , 1980.

eS = E[waiting time for THTT | initial state = S]
Example of pattern occurrence in coin tossing Q. What is the expected waiting time for THTT to occur consecutively? For every state S, define eS = E[waiting time for THTT | initial state = S] It turns out that enull = 18. Prof. Bob Li

eS = E[waiting time till HTHT or THTT | initial state = S]
Example of pattern occurrence in coin tossing Q. What is the expected waiting time for either HTHT or THTT, whichever occurs first? Homework. For every state S, define eS = E[waiting time till HTHT or THTT | initial state = S] In particular, eHTHT = eTHTT = 0. Calculate enull . Two terminal states Prof. Bob Li

P(HTHT eventually occurs | initial state = S]
Probabilities toward multiple terminal states Q. Which pattern between THTT and HTHT is more likely to occur first? For every state S, define pS as P(HTHT eventually occurs | initial state = S] In particular, pHTHT = 1 and pTHTT = 0. The above question becomes: pnull = ? This question does not pertain to the “elapsed time” at any state. The state H eventually leads to HT and hence can be merged into state HT. Similarly we also merge the states T into the state TH. The transition graph: A B

Probabilities toward multiple terminal states
The column vector of pS is an eigenvector of the transition matrix with eigenvalue 1. (Do not confuse this with the limiting distribution, which is a row eigenvector.) Prof. Bob Li

Random-walk gambler A gambler in Macau needs $n for boat fare to go home but has only $i. He gambles $1 at a time and wins with probability p each time. The two terminal states are at ends. Denote pi = P{The gambler will reach $n | starting at $i}. We are to calculate pi for all i.

Calculation by markov chain
pi = p P{Will reach $n | start from $i; initial play wins} + (1p) P{Will reach $n | start from $i; initial play loses} // Conditioning on the outcome of the initial step again // except that we do not use the matrix notation this time. = p P{Will reach $n | start from $i+1} + (1p) P{Will reach $n | start from $i1} Thus, pi = p pi+1 + (1p) pi1 The problem becomes to solve the 2nd-order difference equation with the boundary conditions: p0 = 0 and pn = 1.

pi = p pi+1 + (1p) pi1 p pi + (1p) pi = p pi+1 + (1p) pi1 (1p)(pi  pi1) = p(pi+1  pi) Write xi = pi+1 pi for 0  i < n. (1p) xi1 = p xi This is only a 1st-order difference equation. Thus {xi}0i<N form a geometric progression: xi = x0 Next, we shall make use of the two boundary conditions.

= pi-1 + [ + ] p1 Calculation by markov chain
pi+1 = pi (p1  0) // p0 = 0 pi+1 = pi p1 = [pi p1 ] p // Recursively = pi-1 + [ ] p1 = … = p0 + [1+ … ] p1 // Telescope; p0 = 0 Thus, pi = [1+ … ] p1

It remains to calculate p1 from the boundary condition. Thus take i = n in particular, Later, a martingale will turn this long calculation into an instant one.

Gambler’s Ruin Example. A gambler in Macau starts with $900, needs $1000 to return to HK, gambles $1 at a time on the casino roulette, and wins with probability p = 18/38 at each bet. (Thus, N = 1000 and i = 900.) Then, (1p)/p = 10/9 pi = = (10/9)100 = 2.656105 With the probability , the gambler is doomed. Prof. Bob Li

Remark. Optimal strategy over random walk
Assumptions. The gambler in Macau needs $N to go home but has only $i. At the casino he can bet any integer amount at a time and the probability of winning is 18/38. Since this is not fair gamble to him. It is called a “super martingale,” while the “martingale” concept formalizes fair gamble. Strategy to optimize the chance of going home: Whenever he has $N/2 or less, bet all. Whenever he has more than $N/2, bet the amount that he is short of. Intuition reasoning: Because of the disadvantage, he is better off keeping the gambling process as short as possible. The formal proof of the optimality of the said strategy is beyond the scope of this course. Prof. Bob Li

Skip Simplex algorithm in linear programming
Minimize ct  x // ct = (c1 … cn), xt = (x1 … xn) subject to A  x = b // A is an mn matrix of rank m. // bt = (b1 … bm) and x  0 The optimal value of x is at an extreme point of the feasibility region, a polytope in the n-dim space, and hence has at least n–m components equal to 0. There are extreme points. Label them from 1 to N by the increasing value of the objective function ct  x. The simplex method always moves from an extreme point to a better extreme point.

Efficiency of simplex algorithm in linear programming
The simplex method always moves from an extreme point to a better extreme point. Assumption for the convenience of performance analysis: When the algorithm is at the Lth point, the next extreme point will be equally likely to be any one among the (L–1)st, (L – 2)nd, … , 1st. Thus, Prof. Bob Li

Efficiency of simplex algorithm in linear programming (cont’d)
Let Ti be the number of transitions to go from state i to state 1. In particular, T1 = 0.

Shifting the index, Take the difference between the last two equations. // Approx. by Stirling’s formula

Let c = n/m. Therefore, under the said assumption, the number of simplex-method iterations from the worst start is only Numerical example. If n = 8000 and m = 1000, then c = 8, and

1.4 Time reversed markov chain
Prof. Bob Li

Reverse transition probability
Let Pij denote the transition probability and i the stationary probability of an ergodic markov chain. Assume that the markov chain has been in operation for a long time. Consider the reverse process where n > m  . Define the reverse transition probability Qij At a time m  , Prof. Bob Li

Mnemonic interpretation of j Pji = i Qij
j Pji = stationary rate at which the chain flows from j to i i Qij = stationary rate at which the chain flows backward from i to j Prof. Bob Li

Time reversed markov chain
Theorem. The reverse process where m  , is by itself a markov chain with the transition probability Qij = jPji /i. Prof. Bob Li

at the time m+1 B, A} Prof. Bob Li

The formula Qij = jPji /i calculates:
Reverse transition matrix [Qij]i,j Forward transition matrix [Pij]i,j limiting distribution (… i …) The proposition below offers a way to calculate: Reverse transition matrix [Qij]i,j Forward transition matrix [Pij]i,j limiting distribution (… i …)

Proposition 1. For an irreducible stationary ergodic markov chain with the transition probabilities Pij, if one can find positive numbers i and Qij meeting the equations: // “Irreducible” = can reach any state i from any state j. (a) i Qij = j Pji for all i, j (b) i i = 1 // Normalizer (c) i Qji = 1 for each j // Normalizer then i are the limiting probabilities and Qij are the transition probabilities of the reversed chain. Proof. Summing both sides of (a) over i, In the matrix form, (… i …) [Pij]i,j = (… i …) That is, the row vector (… i …) is an eigenvector of the transition matrix [Pij]i,j with the eigenvalue 1. It must be the limiting distribution because of (b). This, together with (c), shows that Qij are the transition probabilities of the reversed markov chain.

Example of light bulb at the lighthouse
When the light bulb at a very old lighthouse fails during day n, it is replaced at the beginning of the day n+1. Define the integer random variables: L = lifespan of light bulb (rounded up to integer) // Distribution of L is by manufacturer Xn = age of the bulb at the end of day n Thus, {Xn, n = 1, 2, 3, …} is a markov chain. // e.g., {Xn, n = 1, 2, 3, …}={1, 2, 3, 1, 1,…} 1 2 3 4 5 The transition probabilities are:

In the long run, the transition probabilities of the reverse chain are:
When j > 1, clearly Qj,j1 = 1 For all i  1, Q1,i = P{Age of the light bulb yesterday was i | light bulb today is a new one} = P{Age of the light bulb yesterday was i | a light bulb failed yesterday} = P{Age of a light bulb on its day of failing is i} // Yesterday was also in limiting distribution. = P{Lifespan of a light bulb is i} = P{L = i} // Distribution of L is given. By the above Proposition 1, if we shall solve {i} from the equations (a) and (b) below. Then, {i} is the limiting distribution: (a) for all i, j // Need this for j =1 and j  1 (b) i i = // Normalizer (c) i Qji = 1 for each j // Clearly true

Taking j = 1 in (a), for all i  1 From (b), Thus, part of (a) together with (b) uniquely determines {i}. We have yet to verify the remaining part of (a), that is, for all j  1 (and all i  1). Since Qj,j1 = 1, it suffices to verify just for j = i+1 > 1, that is, to show .

Time reversibility Definition. A stationary ergodic markov chain is said to be time reversible if Pij = Qij for all i and j // Recall that i Qij = jPji or, equivalently, for all i and j. // The stationary rate at which the chain flows from i to j // is equal to the stationary rate at which it flows from j to i. An analogy of the stationary rate of transition is the annual trade volume from a nation to another. The limiting distribution is reached when there is overall trade balance for every nation. A time-reversible markov chain is like trade balance between any two nations. This is a stronger statement. Prof. Bob Li

Summing both sides of the equation over i,
Proposition 2. The existence of any nonnegative numbers i summing to 1 such that make the markov chain time reversible. Moreover, such numbers i represent the limiting probabilities. Proof // The proof is similar to and simpler than that of Proposition 1. Summing both sides of the equation over i, // i Pji = 1 = the total probability That is, the row vector (… i …) is an eigenvector of the transition matrix [Pij] with the eigenvalue 1. Hence it is the limiting distribution. Prof. Bob Li

Random walk between two blocking states
The fact that {j} is the limiting distribution corresponds to the red cut. It balances the influx to state i with the outflow from it. In other words, {j} is a (row) eigenvector of the transition matrix with eigenvalue 1. Because the transition graph is a single string of states, the influx to the state group {0, 1, …, i−1} balances with the outflow from it. In other words, the flow from i1 to i and the flow from i to i1 are in equilibrium, as embodied by the simple blue cut. Thus, the process is time reversible. Prof. Bob Li

Random walk between two blocking states
The flow from i1 to i and the flow from i to i1 are in equilibrium. This is equivalent to the eigenvector statement and is mathematically simpler. We now calculate j from it. This is only a 1st-order difference equation, and the boundary condition is the equation of normalization: i i = 1.

Next, calculate 0 from the boundary condition … (Homework)
Taking the product of the above equalities, we can express all i in terms of 0. Next, calculate 0 from the boundary condition … (Homework) Prof. Bob Li

Lecture 4 Sep. 29, 2011  Oct. 6, 2011 Prof. Bob Li

Time-reversibility of random walk 0  M
 The special case when i =  for all i Prof. Bob Li

 The special case when i =   ½ for all i // Here   1, i.e.,   ½ Prof. Bob Li

 The special case when i = ½ for all i . Hence i = 1/(M+1) for all i. // Oscillation between 0 and M leads to uniform stationary distribution. Intuitive interpretation. Consider random walk around the following cycle, in which every line indicates bidirectional transition. 1 2 M

Another way of oscillation
 When i = ½ for all i except that 0 = 1 and M = 0 1 It turns out that i = 1/M for 0 < i < M and 0 = M = 1/2M. Intuitive interpretation. Consider random walk around the cycle below. 1 2 M−1 M Prof. Bob Li

 The special case of a two-urn model M molecules are distributed between two urns. At each transition, one of the M molecules is chosen at random for relocation from its urn to the other. = = = = = = = = Prof. Bob Li

Applying the equation i i = 1,
Model the number of molecules in one urn as a random walk between 0 and M with i = (M  i)/M. Hence, Applying the equation i i = 1, Conclusion. In the limiting distribution, it is as if all M molecules are distributed to the two urns randomly and independently. Homework: Give an intuitive interpretation of this conclusion. Prof. Bob Li

Traveling around a weighted graph
States of the markov chain = nodes in a graph Every edge (i, j) in the graph is associated with a weight wij = wji  0. Transition probabilities Pij = wij / k wik // Weight proportional transition We want to find i such that i i = 1 and because of Proposition 2.  i wij / k wik = j wji / k wjk  i / k wik = j / k wjk // wij = wji  i / k wik = c for some constant c independent of i 1 = i i = c i k wik // Calculate c by normalization.  c = 1 / i k wik  i = k wik / i k wik Prof. Bob Li

Traveling around a weighted graph
Interpretation of formula i = k wik / i k wik Approximating by rational numbers, we may assume that wij are all integers. Replace each edge (i, j) with wij non-weighted edges. Then, a transition from a node i means moving through a randomly selected outgoing edge. The formula says that, regardless of the network topology, the limiting frequency of visiting a node is proportional to the quantity of its adjacent edges. Prof. Bob Li

1.5 Stopping time Prof. Bob Li

// Think of N as the number of customers to a shop
Theorem 1. Let X1, X2, ... , Xn, ... be i.i.d. and N  0 be a discrete random variable that is independent of all Xn. Then, // Think of N as the number of customers to a shop // and Xn as the amount spent by the nth customer. Proof. // Conditioning on N // N is independent of all Xn. // Mean of sum is sum of means. = EX1 EN Prof. Bob LI

1.5.1. Stopping time and Wald’s equation
Q. Roll a die repeatedly until the side “4” shows up. How many rolls does it take on the average? Calculative Solution. Let N  1 represent the waiting time for “4”. P(N = n) = (5/6)n1(1/6) // Geometric1(1/6) EN = 6 Intuitive Solution. Let Xn = 1 or 0 depending on the event for the nth roll to show “4.” // Xn is called the characteristic r.v. of the event. Thus EXn = P{Xn = 1} = 1/6. We speculate that EN is the reciprocal of this probability 1/6. If Theorem 1 in the above could apply, then we could justify the intuition by EN EX1 = EN / 6 However, Theorem 1 requires the independence of N from all Xn, which unfortunately is not the case. So, what to do? We shall strengthen Theorem 1 by relaxing the independence condition in it.

Stopping time Definition. Consider a sequence X1, X2, ... , Xn, ... of random variables. A nonnegative integer random variable N is called a stopping time (or stopping rule) of this sequence if For every n, the event {N  n} is independent of the random variable Xn . // For all n and all x, the events {N  n} and {Xn  x} are independent. // Equivalently, the characteristic r.v. of the event {N  n} is indep. of Xn. Xn represents the outcome of the nth experiment. Right before the nth experiment, you need to decide whether to stop without knowing how Xn will turn out. The decision, though, may depend on the past knowledge on X1, X2, ... , Xn1. Hence, the stopping time N is not independent of the process X1, X2, ... , Xn1, but it abides with causality in life.

Examples of stopping times Roll a die until the side “4” appears.
Gamble $1 at a time at a casino until midnight or losing all money. Gamble $1 at a time at a casino until there is only $k left. Examples of non stopping times Gamble $1 at a time at a casino until right before losing the last $1. Gamble $1 at a time at a casino until right before my luck turns bad. Prof. Bob LI

Wald’s equation Theorem (Wald's Equation). Let X1, X2, ... , Xn, ... be i.i.d. with a stopping time N. If EN < , then Proof. Define the r.v. // Characteristic r.v. of the event {N  n} Observe two things: The stopping time means the indep. between Yn and Xn. XnYn = Xn when N ≥ n = 0 otherwise Hence we can remove the randomness in the upper bound of the summation by Abraham Wald

Wald’s equation = EX1 EN Thus, // Mean of sum is sum of means.
// Xn and Yn are indep. = // Identical distribution // Yn = characteristic r.v. of event {N  n}. = EX1 EN Prof. Bob LI

Example. Gambler’s ruin on casino roulette
Q. A gambler starts with $100 and intends to keep betting $1 on ODD at the roulette until he loses all. How long does this process take on the average? Answer. Let Xn be the net winning on the n-th bet. Then, X1, X2, ... , Xn, ... are i.i.d. and Xn = Thus, EXn = P(Xn = 1)  P(Xn = 1) = 1/19. Let the random variable N represent the number of bets till losing all. Clearly, N is a stopping time for the process X1, X2, ... , Xn, ... By Wald’s equation, Hence EN = 1900 < . Prof. Bob LI

Example of non-positive recurrent markov chain
Theorem. Symmetric 1-dim free random walk is not positive recurrent. // It was shown to be recurrent. Proof. Let Yn be the winning on the n-th game in fair gamble. This defines i.i.d. The symmetric 1-dim free random walk is represented by the markov chain X0, X1, X2, ... , where Xn = j≤n Yj. Let T01 be the waiting time for state 1 given X0 = 0. Clearly T01 is a stopping time for the markov chain. Claim that ET01 = . If ET01 < , Wald's Equation would have yielded the following contradiction: 1 = E [ ] = ET01 EY1 = ET01  0 = 0 Thus ET01 must be . Similarly, ET10 =  = ET1,0. Conditioning on the first move, we have ET00 = 1 + ET10 / 2 + ET1,0 / 2 = 

Revisiting the board game
Let pn = P{ever landing on square #n}. Previously, we found that p6  pn for all n. Q. limnpn = ? Intuitive solution. As n gets very large, the value of pn should have little correlation to n. Thus, limn pn exists. The average value per roll of the die is 7/2. So, 2 out of every 7 squares should be landed. Therefore limn pn = 2/7. Advance the token by rolling a die. Prof. Bob LI

To articulate the intuition in rigor
Denote Xn = outcome of the nth roll Yn = characteristic r.v. of the event {ever landing on n}  pn = P{Yn=1} = EYn Tk = waiting time until the square k is passed = Y1 + Y2 + … + Yk // Count only landed squares.  k+1  X1 + X2 + …  k+6 // Count both landed and passed squares. Clearly Tk is a stopping time for the process X1, X2, … , Xn, … From Wald's Equation, E[X1 + X2 + … ] = ETk EX1 = ETk7/2 k+1  ETk7/2  k+6 2k/7 + 2/7  ETk  2k/7 + 12/7 Since ETk = EY1 + … + EYk + 1 = p1 + … + pk + 1, 2k/7 + 2/7  p1 + p2 + … + pk + 1  2k/7 + 12/7

To articulate the intuition in rigor
2k/7 + 2/7  p1 + p2 + … + pk + 1  2k/7 + 12/7 2/7  5/7k  (p1 + p2 + … + pk)/k  2/7 + 5/7k p1 + p2 + … + pk k limn pn = limk = 2/7 Homework. If we roll two dice instead of just one, show that limn pn = 1/7. Define Wk as the waiting time until the square k is landed or passed. Show that both Tk and Wk are stopping times for the process X1, X2, … , Xn, … Clearly Tk = Y1 + Y2 + … + Yk + 1. How does Wk relate to Y1 + Y2 + … + Yk?

1.6 Martingales Prof. Bob LI

Sum of randomly many i.i.d.
Theorem. When X and Y are independent r.v., E[XY] = EX EY, Cov(X,Y) = 0 and hence Var(X+Y) = Var(X) + Var(Y) Proof. E[XY] // By independence = EX EY Cov(X,Y) = E[(XEX)(YEY)] = E[XY]  E[X(EY)]  E[(EX)Y] + EX EY = E[XY]  (EY)EX  (EX)EY + EX EY = EX EY  EY EX  EX EY+ EX EY = 0

Var(X1+X2+...+Xn) = n Var(X1)
Corollary. Let X1, X2, ... , Xn be i.i.d. Then, Var(X1+X2+...+Xn) = n Var(X1) Proof. Var(X1+X2+...+Xn) = Var(X1) + Var(X2) Var(Xn) // By independence = n Var(X1) // Identical distribution Contrast: Var(nX) = n2 Var(X) // The contrast leads to Central Limit Theorem, // Law of Large Numbers, Chebychev inequality, etc. Proof. Var(nX) = E[(nX)2]  (E[nX])2 = E[n2 X2]  (n EX)2 = n2 E[X2]  n2 (EX)2 = n2 Var(X) Prof. Bob LI

E[Xk+1 | Xk = xk, … , X1 = x1] < xk
Martingale Definition. A process X1, X2, ... , Xn, ... is called a martingale if, for all k, E|Xk| <  and E[Xk+1 | Xk = xk, … , X1 = x1] = xk // This is sometimes abbreviated as E[Xk+1 | Xk, … , X1] = Xk Example. Xk = net cumulative winning after k games in fair gamble. Counterexample. When you gamble on casino roulette, E[Xk+1 | Xk = xk, … , X1 = x1] < xk The process X1, X2, ... , Xn, ... is then called a “super-martingale” by J. Doob ironically. Prof. Bob LI

Martingale = fair gamble
Intuitive definition. A martingale is a stochastic process X0, X1, X2, ... with Xk = cumulative net winning after k games in fair gamble Rigorous definition. A martingale is a stochastic process X0, X1, X2, ... such that E|Xk| <  and for all k. Prof. Bob LI

History of martingale The word martingale came from middle-age French.
It was first related to the concept of probability through a betting system in the 18th century. Paul Levy (1886~1971) formulated the mathematical concept. Prof. Bob LI

Martingale Stopping Theorem
The intuitive fact of E[Net gain in fair gamble upon stopping] = 0 is rigorously formulated as: Martingale Stopping Theorem. (Joseph Doob) Let X0=0, X1, ... , Xn, ... be a martingale and N a stopping time. If E|XN| <  and , then EXN = 0. Proof. Omitted. Prof. Bob LI

An artificial casino of fair gamble
A gambler brings in $1 and bets on Head in fair-coin tossing. Upon the winning every time, he doubles his fortune and then parley the next coin toss. The process continues until he loses. Thus the stopping time N is Geometric1(1/2) distributed. Upon stopping, the net winning of the gambler is $(1), which differs from the initial winning of $0. Therefore, the Martingale Stopping Theorem does not apply here. This example shows that the “lim inf condition” in the Martingale Stopping Theorem is not superfluous.

Gambler’s random walk A gambler in Macau needs $n for boat fare to go home but has only $i. So he gambles $1 at a time and wins with probability p each time. Denote pi = P{Will reach $n | starting at $i}. Recall the long markov-chain calculation for We shall derive this with almost no calculation at all.

Instant calculation by martingale
The random walk is not a martingale. Labels of states are in an arithmetic progression. Deploy a geometric progression instead with the common ratio r = = odds.

The random walk is not a martingale. Labels of states are in an arithmetic progression. Deploy a geometric progression instead with the common ratio r = = odds. By symmetry, we shall assume that p < q. $rk1, the loss is $(rkrk1) From $rk to $rk+1, the gain is $(rk+1rk) = $r(rkrk1) The random walk becomes a martingale X0 = ri, X1, ..., Xn, ...

The time T till the gambler reaches either end is a stopping time. From the Martingale Stopping Theorem, ri = EXT = pi rn + (1 pi)1  // Same solution as before. Prof. Bob LI

Possible applications of this martingale
r = = odds Financial engineering. Take r in this martingale to be the rate of interest, inflation, depreciation, … Information engineering. r can be, for instance, the attenuation of signal strength. Prof. Bob LI

Expected waiting time We have solved by markov chain or by martingale.
Let W = waiting time from state i till reaching either end = a stopping time Q: EW = ? Below we give a solution by a different martingale. Prof. Bob LI

Expected waiting time If p < q, there is an average loss $(qp) per step. If p > q, there is an average gain $(pq) per step. By symmetry, we shall assume the former case. Prof. Bob LI

Expected waiting time EW The average loss per step is $(qp).
To convert the random walk into a martingale, one way is to compensate the gambler by the amount $(qp) per bet. By the martingale stopping theorem, expected net gain upon stopping is 0. E[pi n (qp)W] = i n + (qp) EW = i Thus, EW

Homework. In NBA championship, a “best 4 out of 7” series is played between Miami and Dallas. Assume that the result of every game will be a toss-up. A friendly bookie accepts bets at fair odds on every single game. Starting with an initial capital of $10,000, your mission is to place your bets and eventually double the money if Miami wins the championship. Show that there is one and only one way to accomplish this mission. Hint. Formulate the problem rigorously by the martingale concept. Think of expected value after each game. Prof. Bob LI

Non-binary non-uniform pattern
Roll a special die repeatedly, where P{a} = 1/2 P{b} = 1/3 P{c} = 1/6 The average waiting time for the pattern aba = ? How is this calculated and why so? 14. S.-Y. R. Li, “A martingale approach to the study of occurrence of sequence patterns in repeated experiments,” Annals of Probability, v.8, pp , 1980.

How to calculate average waiting time?
Formula. The average waiting time for the pattern aba is 3 2 + P{a} 1 + P{b} P{a} Calculating 3 bits: 1 = 2 = 3 = a b a 1 a b a 1

How to calculate average waiting time?
Formula. The average waiting time for the pattern aba is 1 Worth 12 0 + 1/2 1 + Worth 2 1/3 = 14 1/2 1 = 2 = 3 = 1 1

Proof by an artificial casino
A gambler arrives with $1 and bets on the pattern aba by rolling the special die: If a appears on the 1st roll, he receives $2 in total. Then, parlay on the 2nd roll. If b appears, he receives $6 in total. Then, parlay on the 3rd roll. If a appears again, he receives $12 in total. Game over. This is fair gamble!

Gambling team enters the artificial casino
Y1 = a = 2 = 1

Y1 = a Y2 = c = 0 = 2 = 0 = 1

Y1 = a Y2 = c Y3 = b Y4 = a Y5 = a Y6 = b Y7 = a = 0 = 0 = 1 = 0 = 0 = 1 = 2 = 2 = 12 = 6 = 1 = 0 = 1 = 2 = 1 Total receipt = regardless of the outcome of die rolling. Net gain of the team upon stopping = – N, where N is the random variable representing the waiting time for aba. Because this is fair gamble, 0 = E[Net gain upon stopping] = 14 – EN  EN = 14

Average waiting time for a pattern
Formula. The average waiting time for the pattern aba is 3rd last gambler has $12 in the end 1 0 + 1/2 Last gambler has $2 in the end 1 + 1/3 = 14 1/2

Lecture 5 Oct. 13, 2011 Prof. Bob Li

Generalizing the example
Formula. For a pattern B = b1b2…bm, calculate j as before. Then, the average waiting time for B is P{Y=b1} 1 + P{Y=b2} m-2 + … m-1 + P{Y=bm} m Prof. Bob Li

The correlation operator 
Example. The average waiting time for the pattern aba is 1 0 + 1/2 1 + 1/3 aba  aba = = 14 1/2 1 = 2 = 3 = 1 1 Prof. Bob LI

Correlation  between two different patterns
3 2 + P{Y=c} 1 + P{Y=b} acab  abc = P{Y=a} a c a b 1 = 2 = 3 = 1 a b c

Artificial casino for betting on pattern abc
At one point, the outcome pattern is acab. Y1 = a Y2 = c Y3 = a Y4 = b = 0 = 0 = 6 = 0 acababc =

Correlation  in the special case of fair-coin toss
1/2 1 + AB = 2 + 3 + 4 = 2  binary(4321) Example. Av. wait for HTHH = HTHHHTHH = 2binary(1001) = 18 Av. wait for HTHH = THTH  THTH = 2binary(1010) = 20 HTHH  THTH = 2  binary(0000) = 0 THTH  HTHH = 2  binary(0101) = 10 Prof. Bob LI

THTH vs. HTHH There are two teams of gamblers called Tortoises and Hares. Before each toss of the fair coin, a new Tortoise joins the casino and bets $1 on the pattern THTH and a new Hare bets $1 on the pattern HTHH. Denote: NA = waiting time for A = THTH in coin tossing NB = waiting time for B = HTHH in coin tossing N = min{NA, NB}, a stopping time for coin-toss process p = P{THTH prevails upon the stopping time N} // 1p = P{HTHH prevails upon the stopping time N} We want to calculate p. Prof. Bob LI

(XN | The pattern THTH prevails) = A  A  A  B
Let Xn be the fortune of Tortoise minus the fortune of Hare after n games. The process X1, X2, X3, … is a martingale, because the gamble is fair. (XN | The pattern THTH prevails) = A  A  A  B (XN | The pattern HTHH prevails) = B  A  B  B By the Martingale Stopping Theorem, 0 = EXN = p(A  A  A  B) + (1p)(B  A  B  B) Hence p : (1p) = (B  B  B  A) : (A  A  A  B) = (18  0) : (20  10) = 9 : 5 Equivalently, p = 9/14. Homework. Compute P(A occurs before B) with a biased coin where P(H) = 0.4. Prof. Bob LI

Alternative calculation of THTH vs. HTHH
BB = ENB = EN + E[NB  N] = EN + p E[NB  N | N = NA] + (1p) E[NB  N | N = NB] = EN + p(BB  AB) + 0  EN = p AB+ (1p) BB Symmetrically, EN = (1p) BA + p AA  p AB+ (1p) BB = p AA+ (1p) BA p : (1p) = (BB  BA) : (AA  AB) = (180) : (2010) = 9 : 5 This alternative calculation can also calculate EN = 90/7. Prof. Bob LI

The general problem T, T, G, C, T A, T, G, C G, G, G, G, G, G, G
Let Y1, Y2, …, Yn, … be i.i.d. representing outcomes of repeated experiments. Given a collection of sequence patterns: T, T, G, C, T A, T, G, C G, G, G, G, G, G, G C, C, A C, A, T, C What is the probability for each pattern to win the race? What is the average waiting time until a winner emerges? Prof. Bob LI

Non-uniform probability distribution
The general problem Non-uniform probability distribution Let Y1, Y2, …, Yn, … be i.i.d. representing outcomes of repeated experiments. Given a collection of sequence patterns: T, T, G, C, T A, T, G, C G, G, G, G, G, G, G C, C, A C, A, T, C What is the probability for each pattern to win the race? What is the average waiting time until a winner emerges? Non-binary: A, T, G, C, … Any number of patterns Uneven lengths Seems a hard problem!

The Main Theorem [Li '80] Main Theorem. Let the random variable N be the waiting time of the i.i.d. process Y1, Y2, …, Yn, … till any of the n competing patterns A1, A2, … , An appears. Denote by pi the winning probability of Ai. Then, // In lifetime, I rarely had such a simple solution to a seemingly formidable problem. Prof. Bob LI

The Main Theorem[Li '80] Main Theorem+. Let the pattern A be present since the beginning. Let the random variable N be the stopping time of the i.i.d. process Y1, Y2, …, Yn, … till any of the n competing patterns A1, A2, … , An appears. Denote by pi the winning probability of Ai. Then, Prof. Bob LI

A lemma to the Main Theorem+
Lemma. Let pattern A be present since the beginning. Then, the expected waiting time for a pattern B is BB  AB The lemma is the special case of Main Theorem+ with only one pattern A1 = B. // The pattern B is not a connected subsequence of pattern A. Prof. Bob LI

[L(A, A)– L(A, B)] : [L(B, B)–L(B, A)]
Earlier results on fair coin-toss John Conway first discovered the integer binary(k…21) in fair coin-toss and called it the leading number L(A, B). His computation algorithms below were quoted by Martin Gardner’ Mathematical games column in [Scientific American, 1974]: The average waiting time for a pattern B = 2  L(B, B). The odds for pattern B to precede pattern A are [L(A, A)– L(A, B)] : [L(B, B)–L(B, A)]  Applying the aforementioned Main Theorem to just one or two patterns in fair coin-toss Prof. Bob LI

Earlier results on coin-toss
Even earlier, William Feller found that, in fair coin-toss: Average waiting time for HHHHHH = 126 Average waiting time for HHTTHH = 70 He also considered the biased coin with P{H} = p and P{T} = q. Through lengthy markov-chain argument: P{A run of m H’s precedes a run of n T’s.}  Applying Main Theorem to particular coin-toss patterns Prof. Bob LI

Mathematical irony The more general the concept,
the more transparent is the theory. Any number of patterns Uneven lengths Non-binary Non-uniform probability distribution

IEG4140 Teletraffic Engineering

Similar presentations

Presentation on theme: "IEG4140 Teletraffic Engineering"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

IEG4140 Teletraffic Engineering

Similar presentations

Presentation on theme: "IEG4140 Teletraffic Engineering"— Presentation transcript:

Similar presentations

About project

Feedback