Download presentation

Presentation is loading. Please wait.

Published byJaylin Ludgate Modified about 1 year ago

1
1 Introduction to Discrete-Time Markov Chain

2
2 Motivation many dependent systems, e.g., inventory across periods state of a machine customers unserved in a distribution system time excellent good fair bad

3
3 Motivation any nice limiting results for dependent X n ’s? no such result for general dependent X n ’s nice results when X n ’s form a discrete-time Markov Chain

4
4 Discrete-Time, Discrete-State Stochastic Process a stochastic process: a sequence of indexed random variables, e.g., {X n }, {X(t)} a discrete-time stochastic process: {X n } a discrete-state stochastic process, e.g., state {excellent, good, fair, bad} set of states {e, g, f, b} {1, 2, 3, 4} {0, 1, 2, 3} state to describe weather {windy, rainy, cloudy, sunny}

5
5 Markov Property a discrete-time, discrete-state stochastic process possesses the Markov property if P{X n+1 = j|X n = i, X n−1 = i n−1,..., X 1 = i 1, X 0 = i 0 } = p ij, for all i 0, i 1, …, i n 1, i n, i, j, n 0 time frame: presence n, future n+1, past {i 0, i 1, …, i n 1 } meaning of the statement: given presence, the past and the future are conditionally independent the past and the future are certainly dependent

6
6 One-Step Transition Probability Matrix p ij 0, i, j 0,

7
7 Example 4-1 Forecasting the Weather state {rain, not rain} dynamics of the system rains today rains tomorrow w.p. does not rain today rains tomorrow w.p. weather of the system across the days, {X n }

8
8 Example 4-3 The Mood of a Person mood {cheerful (C), so-so (S), or glum (G)} cheerful today C, S, or G tomorrow w.p. 0.5, 0.4, 0.1 so-so today C, S, or G tomorrow w.p. 0.3, 0.4, 0.3 glum today C, S, or G tomorrow w.p. 0.2, 0.3, 0.5 X n : mood on the nth day, such that mood {C, S, G} {X n }: a 3-state Markov chain (state 0 = C, state 1 = S, state 2 = G)

9
9 Example 4.5 A Random Walk Model a discrete-time Markov chain of number of states {…, -2, -1, 0, 1, 2, …} random walk: for 0 < p < 1, p i,i+1 = p = 1 − p i,i−1, i = 0, 1,...

10
10 Example 4.6 A Gambling Model each play of a game a gambler gaining $1 w.p. p, and losing $1 o.w. end of the game: a gambler either broken or accumulating $N transition probabilities: p i,i+1 = p = 1 − p i,i−1, i = 1, 2,..., N − 1; p 00 = p NN = 1 example for N = 4 state: X n, the gambler’s fortune after the n play {0, 1, 2, 3, 4}

11
11 Limiting Behavior of Irreducible Chains

12
12 Limiting Behavior of a Positive Irreducible Chain cost of a visit state 1 = $5 state 2 = $8 what is the long-run cost of the above DTMC? 1 2 0.8 0.1 0.9 0.2

13
13 Limiting Behavior of a Positive Irreducible Chain j = fraction of time at state j N: a very large positive integer # of periods at state j j N balance of flow j N i ( i N)p ij j = i i p ij

14
14 Limiting Behavior of a Positive Irreducible Chain j = fraction of time at state j j = i i p ij 1 = 0.9 1 + 0.2 2 2 = 0.1 1 + 0.8 2 linearly dependent normalization equation: 1 + 2 = 1 solving: 1 = 2/3, 2 = 1/3 1 2 0.8 0.1 0.9 C 0.2

15
15 Limiting Behavior of a Positive Irreducible Chain 1 = 0.75 2 + 0.01 3 3 = 0.25 2 1 + 2 + 3 = 1 1 = 301/801, 2 = 400/801, 3 = 100/801 1 2 3 0.25 0.99 1 0.75 0.01

16
16 Limiting Behavior of a Positive Irreducible Chain an irreducible DTMC {X n } is positive there exists a unique nonnegative solution to j : stationary (steady-state) distribution of {X n }

17
17 Limiting Behavior of a Positive Irreducible Chain j = fraction of time at state j j = fraction of expected time at state j average cost c j for each visit at state j random i.i.d. C j for each visit at state j for aperiodic chain:

18
18 Limiting Behavior of a Positive Irreducible Chain 1 = 301/801, 2 = 400/801, 3 = 100/801 profit per state: c 1 = 4, c 2 = 8, c 3 = -2 average profit 1 2 3 0.25 0.99 1 0.75 0.01

19
19 Limiting Behavior of a Positive Irreducible Chain 1 = 301/801, 2 = 400/801, 3 = 100/801 C 1 ~ unif[0, 8], C 2 ~ Geo(1/8), C 3 = -4 w.p. 0.5; and = 0 w.p. 0.5 E(C 1 ) = 4, E(C 2 ) = 8, E(C 3 ) = -2 average profit 1 2 3 0.25 0.99 1 0.75 0.01

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google