1. Section 8: Markov Decision Process

2. The Clinic’s Problem You are a doctor and run a clinic which opens 2 hours a day Each hour you will have one patient: 80% of the time (s)he has flu, 20% of the time (s)he has Ebola You can either send them home or to the hospital It’s better when patients survive at home than if they survive at the hospital. But we definitely don’t want them to die! A patient with the flu has 90% chance to survive at home and will survive at the hospital for sure A patient with Ebola has 50% chance to survive at home and will survive at the hospital for sure.

3. Home or Hospital? For each patient who visits your clinic in a day, should you send him/her home or to the hospital?

4. Modeling in MDP

5. Model (1) States: a four-element tuple [N, C, PA, PS] - N: number of patients visited so far - C: condition of the current patient - PA: the action on the previous patient - PS: whether the previous patient survives Actions: - send home, send to hospital, null action Reward Model: - If s(PA)=home and s(PS)=1, R(s, a)=2 - If s(PA)=home and s(PS)=0, R(s, a)=0 - If s(PA)=hospital and s(PS)=1, R(s, a)=1

6. Model (2) [0, F, n/a, n/a] [0, E, n/a, n/a] home [1, F, home, 1] [1, E, home, 1] … [1, F, home, 0] [1, E, home, 0] … [2, n/a, hospital, 1] [2, n/a, home, 1] [2, n/a, hospital, 0] [2, n/a, home, 0] 0.8*0.9=0.72 0.8*0.1=0.08 0.2*0.9=0.18 0.2*0.1=0.02 Reward=0! hospital 1 0 Reward=2! home 0.9 0.1 Reward=2! Finish null Reward=1!

7. Eat ice-cream or take vitamins??? States: Healthy (H) or Sick (S) Actions: Ice-cream (IC) or Vitamins (V) Transitions: – Ice cream Vitamins Rewards: HS H0.70.3 S01 HS H10 S0.90.1 HS IC105 V00

8. Value Iteration

9. kQ k (H,IC)Q k (H,V)Q k (S,IC)Qk(S,V)Qk(S,V) *k(H)*k(H) *k(S)*k(S) Vk(H)Vk(H)Vk(S)Vk(S) 000 110050IC 105 299.58.55IC 17.659.5 17.65 315.8813.5515.15ICV23.6815.15 23.68