1. ST3236: Stochastic Process Tutorial 4 TA: Mar Choong Hock Email: g0301492@nus.edu.sg Exercises: 5

2. Question 1 An urn initially contains a single red and single green ball. A ball is drawn at random, removed and replaced by a ball of the opposite color and this procedure repeats so that there are always two balls in the urn. Let X n be the number of red balls in the urn after n draws, with X 0 = 1. Specify the transitions probabilities for MC {X}.

3. Question 1 Case (X n =0): Both balls are green. One ball will certainly be replaced with red after a ball is drawn. P(X n+1 = 1 | X n = 0) = 1, P(X n+1 = 0 | X n = 0) = 0, P(X n+1 = 2 | X n = 0) = 0 Case (X n =2): Both balls are red. One ball will certainly be replaced with green after a ball is drawn. P(X n+1 = 1 | X n = 2) = 1, P(X n+1 = 0 | X n = 2) = 0, P(X n+1 = 2 | X n = 2) = 0

4. Question 1 Case (X n = 1): One ball is red. Outcome dependent on the colour of the ball drawn. Note: P(Green is drawn|X n =1) = P (Red is drawn|X n =1) = 0.5 P(X n+1 = 0 | X n = 1) = P(Red is drawn|X n = 1) = 0.5 P(X n+1 = 2 | X n = 1) = P(Green is drawn|X n = 1) = 0.5 P(X n+1 = 1 | X n = 1) = 0

5. Question 1 The transition matrix is given as follows:

6. Question 2 Find the mean time to reach state 3 starting from state 0 for the MC whose transition probability matrix is

7. Question 2 Let T = min{n : X n = 3} and v i = E(T | X 0 = i). The mean time to reach state 3 starting from state 0 is v 0. We apply first step analysis.

8. Question 2 Therefore, v 0 = 1 + 0.4v 0 + 0.3v 1 + 0.2v 2 + 0.1v 3 v 1 = 1 + 0v 0 + 0.7v 1 + 0.2v 2 + 0.1v 3 v 2 = 1 + 0v 0 + 0v 1 + 0.9v 2 + 0.1v 3 v 3 = 0 Solving the equations, we have v 0 = 10.

9. Question 3 Consider the MC with transition probability matrix (a) Starting in state 1, determine the probability that the MC ends in state 0 (b) Determine the mean time to absorption.

10. Question 3a Let T = min{n : X n = 0 and X n = 2}, u i = P(X T = 0|X 0 = i) and v i = E(T|X 0 = i). u 0 = 1 u 1 = 0.1u 0 + 0.6u 1 + 0.3u 2 u 2 = 0 we have u 1 = 0.25.

11. Question 3b Let T = min{n : X n = 0 and X n = 2}, u i = P(X T = 0|X 0 = i) and v i = E(T|X 0 = i). v 0 = 0 v 1 = 1 + 0.1v 0 + 0.6v 1 + 0.3v 2 v 2 = 0 we have v 1 = 2.5.

12. Question 4 Consider the MC with transition probability matrix (a) Starting in state 1, determine the probability that the MC ends in state 0 (b) Determine the mean time to absorption.

13. Question 4a Let T = min{n : X n = 0 and X n = 2}, u i = P(X T = 0|X 0 = i) and v i = E(T|X 0 = i). u 0 = 1 u 1 = 0.1u 0 + 0.6u 1 + 0.1u 2 + 0.2u 3 u 2 = 0.2u 0 + 0.3u 1 + 0.4u 2 + 0.1u 3 u 3 = 0 we have, u 1 = 0.3810, u 2 = 0.5238 Starting in state 1, the probability that the MC ends in state 0 is u 1 = 0.3810

14. Question 4b Let T = min {n : X n = 0 and X n = 2}, u i = P(X T = 0|X 0 = i) and v i = E(T|X 0 = i). v 0 = 0 v 1 = 1 + 0.1v 0 + 0.6v 1 + 0.1v 2 + 0.2v 3 v 2 = 1 + 0.2v 0 + 0.3v 1 + 0.4v 2 + 0.1v 3 v 3 = 0 we have v 1 = v 2 = 3.33.

15. Question 5 A coin is tossed repeatedly until two successive heads appear. Find the mean number of tosses required. [Hint: Let X n be the cumulative number of successive heads. Then the state space is 0,1,2 before stop]

16. Question 5 – Method 1 Let Y n be the outcome {H, T} of each toss and (Y n-1, Y n ) denotes the sample point for the sucessive tosses. There are 4 possible sample points.

18. Question 5 – Method 1 The transition probability matrix is

19. Question 5 – Method 1 Let v i denote the mean time to each state 3 starting from state i. By the first step analysis, we have the following equations: v 0 = 1 + 0.5v 0 + 0.5v 1 v 1 = 1 + 0.5v 2 + 0.5v 3 v 2 = 1 + 0.5v 0 + 0.5v 1 v 3 = 0 Therefore, v 0 = 6 (v 1 = 4, v 2 = 6)

20. Question 5 – Method 2 Let X n be the cumulative number of successive heads. The 3-state state space now is {0,1,2}. Example: n01234 YnYn THTHH XnXn 01012

21. Question 5 – Method 2 Case (X n = 0) :Previous two tosses are tails Or a head followed by tail. Note: P(Head) = P(Tail) = 0.5 P(X n+1 = 1 | X n = 0) = P(Head) = 0.5, P(X n+1 = 0 | X n = 0) = P(Tail) = 0.5, P(X n+1 = 2 | X n = 0) = 0 Case (X n = 1): A tail is followed by a head P(X n+1 = 2 | X n = 1) = P(Head) = 0.5 P(X n+1 = 0 | X n = 1) = P(Tail) = 0.5 P(X n+1 = 1 | X n = 1) = 0

22. Question 5 – Method 2 Case (X n = 2): Previous two tosses are heads. We make this state absorbing. P(X n+1 = 1 | X n = 2) = 0 P(X n+1 = 2 | X n = 2) = 1, P(X n+1 = 0 | X n = 2) = 0

23. Question 5 – Method 2 The transition probability matrix is

24. Question 5 – Method 2 Let v i denote the mean time to each state 2 starting from state i. By the first step analysis, we have the following equations: v 0 = 1 + 0.5v 0 + 0.5v 1 v 1 = 1 + 0.5v 0 + 0.5v 2 v 2 = 0 Thus, we have, v 0 = 6, v 1 = 4