1. ST3236: Stochastic Process Tutorial 5
TA: Mar Choong Hock
Exercises: 6

2. Question 1 Consider the MC with transition probability matrix
Starting in state 1, determine the mean time that the
process spends in state 1 prior to absorption and
the mean time that the process spends in state 2
prior to absorption. Verify that the sum of these is
the mean time to the absorption.

3. Question 1
Denote by wi1 the mean time the process spends in state 1 starting in state i prior to the absorption.
We have,
w01 = 0
w11 = w w w w31
w21 = w w w w31
w31 = 0
The solution is
w01 = 0, w11 = , w21 = , w31 = 0.

4. Question 1 Similarly, denote by wi2 the mean time the process
spends in state 2 starting in state i prior to the
absorption. We have,
w02 = 0
w12 = w w w w32
w22 = w w w w32
w32 = 0
The solution is
w02 = 0, w12 = , w22 = , w32 = 0.

5. Question 1
Denote by vi the mean time to the absorption starting in state i prior to the absorption.
We have,
v0 = 0
v1 = v v v v3
v2 = v v v v3
v3 = 0
The solution is v0 = 0, v1 = , v2 = ,
v3 = 0.
We have verified that,
v1 = w11 + w12

6. Question 2 Consider the MC with transition probability matrix
Starting in state 1, determine the mean time that the
process spends in state 1 prior to absorption and
the mean time that the process spends in state 2
prior to absorption. Verify that the sum of these is
the mean time to the absorption.

7. Question 2 Denote by wi1 the mean time the process spends in
state 1 starting in state i prior to the absorption.
We have,
w01 = 0
w11 = w w w w31
w21 = w w w w31
w31 = 0
The solution is,
w01 = 0, w11 = 1.290, w21 = , w31 = 0.

8. Question 2 Similarly, denote by wi2 the mean time the process
spends in state 2 starting in state i prior to the
absorption. We have,
w02 = 0
w12 = w w w w32
w22 = w w w w32
w32 = 0
The solution is
w02 = 0, w12 = , w22 = , w32 = 0.

9. Question 2 Denote by vi the mean time to the absorption
starting in state i prior to the absorption.
We have,
v0 = 0
v1 = v v v v3
v2 = v v v v3
v3 = 0
The solution is v0 = 0, v1 = 1.613, v2 = ,
v3 = 0.
We have verified that,
v1 = w11 + w12

10. Question 3 Consider the MC in question 1. Starting in state 1,
determine the probability that the process is
absorbed into state 0. Compare this with the (1,0)th
entry in the matrix powers P2, P4, P8 and P16.

11. Question 3 Denote by ui the probability that the MC is absorbed
by 0 starting in state i.
We have,
u0 = 1
u1 = 0.1u u u u3
u2 = 0.1u u u u3
u3 = 0
The solution is,
u0 = 1, u1 = , u2 = , u3 = 0.

12. Compare:
u1 =

13. Question 3 By definition,
Consider a (4 x 4) transition probability matrix,

14. Question 3
But for our case, p00 = 1, p03 = 0.

15. Question 3-Optional Let:
F(t) be the set of t-step first passage paths from state 1 to state 0
G(n-t) be the set of (n-t)-step paths from state 0 to state 0
H(t) be the set of paths that is formed jointly by F(t) followed by G(n-t). Note: paths are n-step.

16. Question 3-Optional

17. Question 3-Optional Let L(n) be the set of n-step paths from state 1
to state 0. s.t.

18. Question 3-Optional
f1,0(t) is the t-step first passage probability from
state 1 to state 0.
If state 0 is an absorbing state,
Also, trivially,

19. Question 3-Optional

20. Question 4
Which of the following MC is regular: a) b)

21. Question 4 a) YES, because (all entries are greater than 0)
b) NO, because it has absorbing states