Download presentation

Presentation is loading. Please wait.

Published byCristina Hogard Modified over 2 years ago

1
1 OR II GSLM 52800

2
2 Outline introduction to discrete-time Markov Chain introduction to discrete-time Markov Chain problem statement long-term average cost pre unit time solving the MDP by linear programming

3
3 States of a Machine StateCondition 0Good as new 1Operable – minor deterioration 2Operable – major deterioration 3Inoperable – output of unacceptable quality

4
4 Transition of States State0123 007/81/16 103/41/8 2001/2 30001

5
5 Possible Actions DecisionActionRelevant States 1Do nothing0, 1, 2 2 Overhaul (return to state 1) 2 3 Replace (return to state 0) 1, 2, 3

6
6 Problem adopting different collections of actions leading to different long-term average cost per unit time problem: to find the policy that minimizes the long-term average cost per unit time

7
7 Costs of Problem cost of defective items state 0: 0; state 1: 1000; state 2: 3000 cost of replacing the machine = 4000 cost of losing production in machine replacement = 2000 cost of overhauling (at state 2) = 2000

8
8 Policy R d : Always Replace When State 0 half of the time at state 0, with cost 0 half of the time at other states, all with cost 6000, because of machine replacement average cost per unit time = 3000 0 1 2 3 1/16 7/8 1 1 1 1/16

9
9 Long-Term Average Cost of a Positive, Irreducible Discrete-time Markov Chain a positive, irreducible discrete-time Markov chain with M+1 states, 0, …, M only M of the balance eqt plus the normalization eqt

10
10 Policy R a : Replace at Failure but Otherwise Do Nothing 0 1 23 1/16 7/8 1 1/2 1 1/16 3/4 1/8 1/2

11
11 Policy R b : Replace in State 3, and Overhaul in State 2 0 1 2 3 1/16 7/8 1 1 1/16 3/4 1/8 1

12
12 Policy R c : Replace in States 2 and 3 0 1 2 3 1/16 7/8 1 1 1/16 3/4 1/8 1 1

13
13 Problem R b, i.e., replacing in State 3 and overhauling in State 2 in this case the minimum-cost policy is R b, i.e., replacing in State 3 and overhauling in State 2 question: Is there any efficient way to find the minimum cost policy if there are many states and different types of actions? impossible to check all possible cases

14
14 Linear Programming Approach for an MDP let D ik be the probability of adopting decision k at state i i be the stationary probability of state i y ik = P(state i and decision k) C ik = the cost of adopting decision k at state i

15
15 Linear Programming Approach for an MDP

16
16 Linear Programming Approach for an MDP at optimal, D ik = 0 or 1, i.e., a deterministic policy is used

17
17 Linear Programming Approach for an MDP actions possibly to adopt at state 0: do nothing (i.e., k = 1) 1: do nothing or replace (i.e., k = 1 or 3) 2: do nothing, overhaul, or replace (i.e., k = 1, 2, or 3) 3: replace (i.e., k = 3) variables: y 01, y 11, y 13, y 21, y 22, y 23, and y 33

18
18 Linear Programming Approach for an MDP Stat e 0123 007/81/16 103/41/8 2001/2 30001

19
19 Linear Programming Approach for an MDP solving, y 01 = 2/21, y 11 = 5/7, y 13 = 0, y 21 = 0, y 22 = 2/21, y 23 = 0, y 33 = 2/21 optimal policy at state 0: do nothing state 1: do nothing state 2: overhaul state 3: replace

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google