1. 1 Introduction to Discrete-Time Markov Chain

2. 2 Motivation  many dependent systems, e.g.,  inventory across periods  state of a machine  customers unserved in a distribution system time excellent good fair bad

3. 3 Motivation  any nice limiting results for dependent X n ’s?  no such result for general dependent X n ’s  nice results when X n ’s form a discrete-time Markov Chain

4. 4 Discrete-Time, Discrete-State Stochastic Process  a stochastic process: a sequence of indexed random variables, e.g., {X n }, {X(t)}  a discrete-time stochastic process: {X n }  a discrete-state stochastic process, e.g.,  state  {excellent, good, fair, bad}  set of states  {e, g, f, b}  {1, 2, 3, 4}  {0, 1, 2, 3}  state to describe weather  {windy, rainy, cloudy, sunny}

5. 5 Markov Property  a discrete-time, discrete-state stochastic process possesses the Markov property if  P{X n+1 = j|X n = i, X n−1 = i n−1,..., X 1 = i 1, X 0 = i 0 } = p ij, for all i 0, i 1, …, i n  1, i n, i, j, n  0  time frame: presence n, future n+1, past {i 0, i 1, …, i n  1 }  meaning of the statement: given presence, the past and the future are conditionally independent  the past and the future are certainly dependent

6. 6 One-Step Transition Probability Matrix  p ij  0, i, j  0,

7. 7 Example 4-1 Forecasting the Weather  state  {rain, not rain}  dynamics of the system  rains today  rains tomorrow w.p.   does not rain today  rains tomorrow w.p.   weather of the system across the days, {X n }

8. 8 Example 4-3 The Mood of a Person  mood  {cheerful (C), so-so (S), or glum (G)}  cheerful today  C, S, or G tomorrow w.p. 0.5, 0.4, 0.1  so-so today  C, S, or G tomorrow w.p. 0.3, 0.4, 0.3  glum today  C, S, or G tomorrow w.p. 0.2, 0.3, 0.5  X n : mood on the nth day, such that mood  {C, S, G}  {X n }: a 3-state Markov chain (state 0 = C, state 1 = S, state 2 = G)

9. 9 Example 4.5 A Random Walk Model  a discrete-time Markov chain of  number of states {…, -2, -1, 0, 1, 2, …}  random walk: for 0 < p < 1,  p i,i+1 = p = 1 − p i,i−1, i = 0,  1,...

10. 10 Example 4.6 A Gambling Model  each play of a game a gambler gaining $1 w.p. p, and losing $1 o.w.  end of the game: a gambler either broken or accumulating $N  transition probabilities:  p i,i+1 = p = 1 − p i,i−1, i = 1, 2,..., N − 1; p 00 = p NN = 1  example for N = 4  state: X n, the gambler’s fortune after the n play  {0, 1, 2, 3, 4}

11. 11 Limiting Behavior of Irreducible Chains

12. 12 Limiting Behavior of a Positive Irreducible Chain  cost of a visit  state 1 = $5  state 2 = $8  what is the long-run cost of the above DTMC? 1 2 0.8 0.1 0.9 0.2

13. 13 Limiting Behavior of a Positive Irreducible Chain   j = fraction of time at state j  N: a very large positive integer  # of periods at state j   j N  balance of flow   j N   i (  i N)p ij   j =  i  i p ij

14. 14 Limiting Behavior of a Positive Irreducible Chain   j = fraction of time at state j   j =  i  i p ij   1 = 0.9  1 + 0.2  2   2 = 0.1  1 + 0.8  2  linearly dependent  normalization equation:  1 +  2 = 1  solving:  1 = 2/3,  2 = 1/3 1 2 0.8 0.1 0.9 C 0.2

15. 15 Limiting Behavior of a Positive Irreducible Chain   1 = 0.75  2 + 0.01  3   3 = 0.25  2   1 +  2 +  3 = 1   1 = 301/801,  2 = 400/801,  3 = 100/801 1 2 3 0.25 0.99 1 0.75 0.01

16. 16 Limiting Behavior of a Positive Irreducible Chain  an irreducible DTMC {X n } is positive  there exists a unique nonnegative solution to    j : stationary (steady-state) distribution of {X n }

17. 17 Limiting Behavior of a Positive Irreducible Chain   j = fraction of time at state j   j = fraction of expected time at state j  average cost  c j for each visit at state j  random i.i.d. C j for each visit at state j  for aperiodic chain:

18. 18 Limiting Behavior of a Positive Irreducible Chain   1 = 301/801,  2 = 400/801,  3 = 100/801  profit per state: c 1 = 4, c 2 = 8, c 3 = -2  average profit 1 2 3 0.25 0.99 1 0.75 0.01

19. 19 Limiting Behavior of a Positive Irreducible Chain   1 = 301/801,  2 = 400/801,  3 = 100/801  C 1 ~ unif[0, 8], C 2 ~ Geo(1/8), C 3 = -4 w.p. 0.5; and = 0 w.p. 0.5  E(C 1 ) = 4, E(C 2 ) = 8, E(C 3 ) = -2  average profit 1 2 3 0.25 0.99 1 0.75 0.01