Presentation is loading. Please wait.

Presentation is loading. Please wait.

Modeling Long Term Care and Supportive Housing Marisela Mainegra Hing Telfer School of Management University of Ottawa Canadian Operational Research Society,

Published byNigel Malone Modified over 8 years ago

Similar presentations

Presentation on theme: "Modeling Long Term Care and Supportive Housing Marisela Mainegra Hing Telfer School of Management University of Ottawa Canadian Operational Research Society,"— Presentation transcript:

1 Modeling Long Term Care and Supportive Housing Marisela Mainegra Hing Telfer School of Management University of Ottawa Canadian Operational Research Society, May 18, 2011

2 Outline  Long Term Care and Supportive Housing  Queueing Models  Dynamic Programming Model  Approximate Dynamic Programming

3 μ LTC, C LTC Community Hospital LTC λ C λHλH λRCλRC λ RH LTC problem Goal: Hospital level below a given threshold Community waiting times below 90 days

4 LTC previous results  MDP model determined a threshold policy for the Hospital but it did not take into account community demands  Simulation Model determined that current capacity is insufficient to achieve the goal

5 Queueing Model Station LTC: M/M/C LTC Station H_renege: M/M/∞ μ LTC, C LTC λ H-LTC λ C-LTC λ LTC λ RH λRCλRC Hospital λHλH Community λ C LTC H_reneg e μ RH, 

6 Queueing Model Station LTC: M/M/C LTC <1 steady state: The probability that no patients are in the system: The average number of patients in the waiting line: The average time a client spends in the waiting line: The number of patients from the Hospital that are in the queue for LTC (L qH-LTC ).

7 Queueing Model Station H_renege: M/M/∞  The average number of patients in the system is

8 Queueing Model Data analysis  Data on all hospital demand arriving to the CCAC from April 1st, 2006 to May 15th, 2009.  ρ LTC = 1.6269 for current capacity C LTC = 4530  To have ρ LTC 7370.08, 2841 (62.71%) more beds than the current capacity.  With CLTC > 7370 we apply the formulas.  Given a threshold T for the hospital patients and the number Lq LTC of total patients waiting to go to LTC, what we want is to determine the capacity CLTC in LTC such as:

9 Queueing Model Results  19 iterations of capacity values  Goal achieved with capacity 7389, the average waiting time is 31 days and the average amount of Hospital patients waiting in the queue is 130 ( T=134).  This required capacity is 2859 (63.1%) more than the current capacity.

10 Queueing Model with SH λ RH Hospital μ LTC, C LTC λ H-LTC λRCλRC λHλH Community λ C LTC H_renege λ C-SH λ H-SH SH μ SH, C SH λ SH-LTC λ C-LTC μ RH, 

11 Queueing Model with SH Results  Required capacity in LTC is 6835, 2305 (50.883%) more beds than the current capacity (4530).  Required capacity in SH is 1169.  With capacity values at LTC: 6835 and at SH: 1169 there are 133.9943 (T= 134) Hospital Patients waiting for care (for LTC: 110.3546, reneging: 22.7475, for SH: 0.89229), and Community Patients wait for care in average (days) at LTC: 34.8799, and at SH: 3.2433.

12 Semi-MDP Model State space: Action space: Transition time: Transition probabilities: Immediate reward: Optimal Criterion: S = {(D H_LTC, D H_SH, D C_LTC, D C­_SH, D SH_LTC, C LTC, C SH, p) } A = {0,..,max(TC LTC,TC SH )} d(s,a) = Pr(s,a,s’) = r(s,a) = Total expected discounted reward

13 Approximate Dynamic programming γ: discount factor find π : S A that maximizes the state-action value function Goal Bellman: there exists Q* optimal: Q* =maxQ(s,a) and the optimal policy π*

14 state action Reinforcement Reinforcement Learning

15 RL: environment transition probabilities reward function action next state, immediate reward ENVIROMMENT state

16 RL: Agent Knowledge: Q(s,a) exploratory Learning: update Q-values state Knowledge representation (FA) Backup table Neural networkNeural... Watkins QL Sarsa ( )... reward action Learning method Behavior

17 QL: parameters  θ: number of hidden neurons.  T: number of iterations of the learning process.   0 : initial value of the learning rate.   0 : initial value of the exploration rate.  Learning-rate decreasing function.  Exploration-rate decreasing function.

18 QL: algorithm exploration vs/ exploitation Learning and exploration rates θ (T, T

19 QL: tuning parameters (observed regularities) 1.(θ,  )-scheme: T=  10 4,  0 = 10 -3,  0 =1, T  =  10 3, T  =v10 3, v  [1,..  ]. PR(θ,  ): best performance with (θ,  )-scheme 2.PR(θ,  ) monotically increase respect  until certain value  (θ) 3.PR(θ,  ) monotically increase respect θ until certain value θ(  ) 4.  (θ) and θ(  ) depend on the problem instance

20 QL: tuning parameters (methodology: learning schedule given PR Heu ) 1.∆θ =50, θ =0, PR θ =0,  =0, v best =1, 2.While PR θ <PR heu or no-stop 1.θ = θ + ∆θ, PR best =0 2.While PR best ≥ PR  1.  =  +1, T=  10 4, T  =  10 3 2.PR  =PR best, 3.For v= v best to  T  =v10 3 PR[v]=Q-Learning(T, θ, 10 -3, 1, T , T  ) 4.[PR best,v best ]=max(PR) 3.PR θ = PR 

21 Discussion  For given capacities solve the SMDP with QL  Model other LTC complexities: different facilities and room accommodations, client choice and level of care

22 Thank you for your attention  Questions?

23 Neural Network for Q(s,a)

Download ppt "Modeling Long Term Care and Supportive Housing Marisela Mainegra Hing Telfer School of Management University of Ottawa Canadian Operational Research Society,"

Similar presentations

Value Iteration & Q-learning CS 5368 Song Cui. Outline Recap Value Iteration Q-learning.

Value Iteration & Q-learning CS 5368 Song Cui. Outline Recap Value Iteration Q-learning.
A Markov Decision Model for Determining Optimal Outpatient Scheduling Jonathan Patrick Telfer School of Management University of Ottawa.

A Markov Decision Model for Determining Optimal Outpatient Scheduling Jonathan Patrick Telfer School of Management University of Ottawa.
1 Reinforcement Learning Problem Week #3. Figure reproduced from the figure on page 52 in reference [1] 2 Reinforcement Learning Loop state Agent Environment.

1 Reinforcement Learning Problem Week #3. Figure reproduced from the figure on page 52 in reference [1] 2 Reinforcement Learning Loop state Agent Environment.
INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, 2014 Lecture.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.
Reinforcement learning (Chapter 21)

Reinforcement learning (Chapter 21)
1 Monte Carlo Methods Week #5. 2 Introduction Monte Carlo (MC) Methods –do not assume complete knowledge of environment (unlike DP methods which assume.

1 Monte Carlo Methods Week #5. 2 Introduction Monte Carlo (MC) Methods –do not assume complete knowledge of environment (unlike DP methods which assume.
1 Temporal-Difference Learning Week #6. 2 Introduction Temporal-Difference (TD) Learning –a combination of DP and MC methods updates estimates based on.

1 Temporal-Difference Learning Week #6. 2 Introduction Temporal-Difference (TD) Learning –a combination of DP and MC methods updates estimates based on.
COSC 878 Seminar on Large Scale Statistical Machine Learning 1.

COSC 878 Seminar on Large Scale Statistical Machine Learning 1.
ETHEM ALPAYDIN © The MIT Press, 2010 Lecture Slides for.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.
Reinforcement learning

Reinforcement learning
Università di Milano-Bicocca Laurea Magistrale in Informatica Corso di APPRENDIMENTO E APPROSSIMAZIONE Lezione 6 - Reinforcement Learning Prof. Giancarlo.

Università di Milano-Bicocca Laurea Magistrale in Informatica Corso di APPRENDIMENTO E APPROSSIMAZIONE Lezione 6 - Reinforcement Learning Prof. Giancarlo.
Reinforcement Learning Tutorial

Reinforcement Learning Tutorial
Blackjack as a Test Bed for Learning Strategies in Neural Networks A. Perez-Uribe and E. Sanchez Swiss Federal Institute of Technology IEEE IJCNN'98.

Blackjack as a Test Bed for Learning Strategies in Neural Networks A. Perez-Uribe and E. Sanchez Swiss Federal Institute of Technology IEEE IJCNN'98.
Outline MDP (brief) –Background –Learning MDP Q learning Game theory (brief) –Background Markov games (2-player) –Background –Learning Markov games Littman’s.

Outline MDP (brief) –Background –Learning MDP Q learning Game theory (brief) –Background Markov games (2-player) –Background –Learning Markov games Littman’s.
Reinforcement Learning Mitchell, Ch. 13 (see also Barto & Sutton book on-line)

Reinforcement Learning Mitchell, Ch. 13 (see also Barto & Sutton book on-line)
Cooperative Q-Learning Lars Blackmore and Steve Block Expertness Based Cooperative Q-learning Ahmadabadi, M.N.; Asadpour, M IEEE Transactions on Systems,

Cooperative Q-Learning Lars Blackmore and Steve Block Expertness Based Cooperative Q-learning Ahmadabadi, M.N.; Asadpour, M IEEE Transactions on Systems,
1 Hybrid Agent-Based Modeling: Architectures,Analyses and Applications (Stage One) Li, Hailin.

1 Hybrid Agent-Based Modeling: Architectures,Analyses and Applications (Stage One) Li, Hailin.
1 Kunstmatige Intelligentie / RuG KI2 - 11 Reinforcement Learning Johan Everts.

1 Kunstmatige Intelligentie / RuG KI Reinforcement Learning Johan Everts.
Reinforcement Learning: Learning algorithms Yishay Mansour Tel-Aviv University.

Reinforcement Learning: Learning algorithms Yishay Mansour Tel-Aviv University.
More RL. MDPs defined A Markov decision process (MDP), M, is a model of a stochastic, dynamic, controllable, rewarding process given by: M = 〈 S, A,T,R.

More RL. MDPs defined A Markov decision process (MDP), M, is a model of a stochastic, dynamic, controllable, rewarding process given by: M = 〈 S, A,T,R.

Similar presentations

Ads by Google