Modeling Long Term Care and Supportive Housing Marisela Mainegra Hing Telfer School of Management University of Ottawa Canadian Operational Research Society, May 18, 2011

Outline  Long Term Care and Supportive Housing  Queueing Models  Dynamic Programming Model  Approximate Dynamic Programming

μ LTC, C LTC Community Hospital LTC λ C λHλH λRCλRC λ RH LTC problem Goal: Hospital level below a given threshold Community waiting times below 90 days

LTC previous results  MDP model determined a threshold policy for the Hospital but it did not take into account community demands  Simulation Model determined that current capacity is insufficient to achieve the goal

Queueing Model Station LTC: M/M/C LTC Station H_renege: M/M/∞ μ LTC, C LTC λ H-LTC λ C-LTC λ LTC λ RH λRCλRC Hospital λHλH Community λ C LTC H_reneg e μ RH, 

Queueing Model Station LTC: M/M/C LTC <1 steady state: The probability that no patients are in the system: The average number of patients in the waiting line: The average time a client spends in the waiting line: The number of patients from the Hospital that are in the queue for LTC (L qH-LTC ).

Queueing Model Station H_renege: M/M/∞  The average number of patients in the system is

Queueing Model Data analysis  Data on all hospital demand arriving to the CCAC from April 1st, 2006 to May 15th,  ρ LTC = for current capacity C LTC = 4530  To have ρ LTC , 2841 (62.71%) more beds than the current capacity.  With CLTC > 7370 we apply the formulas.  Given a threshold T for the hospital patients and the number Lq LTC of total patients waiting to go to LTC, what we want is to determine the capacity CLTC in LTC such as:

Queueing Model Results  19 iterations of capacity values  Goal achieved with capacity 7389, the average waiting time is 31 days and the average amount of Hospital patients waiting in the queue is 130 ( T=134).  This required capacity is 2859 (63.1%) more than the current capacity.

Queueing Model with SH λ RH Hospital μ LTC, C LTC λ H-LTC λRCλRC λHλH Community λ C LTC H_renege λ C-SH λ H-SH SH μ SH, C SH λ SH-LTC λ C-LTC μ RH, 

Queueing Model with SH Results  Required capacity in LTC is 6835, 2305 (50.883%) more beds than the current capacity (4530).  Required capacity in SH is  With capacity values at LTC: 6835 and at SH: 1169 there are (T= 134) Hospital Patients waiting for care (for LTC: , reneging: , for SH: ), and Community Patients wait for care in average (days) at LTC: , and at SH:

Semi-MDP Model State space: Action space: Transition time: Transition probabilities: Immediate reward: Optimal Criterion: S = {(D H_LTC, D H_SH, D C_LTC, D C­_SH, D SH_LTC, C LTC, C SH, p) } A = {0,..,max(TC LTC,TC SH )} d(s,a) = Pr(s,a,s’) = r(s,a) = Total expected discounted reward

Approximate Dynamic programming γ: discount factor find π : S A that maximizes the state-action value function Goal Bellman: there exists Q* optimal: Q* =maxQ(s,a) and the optimal policy π*

state action Reinforcement Reinforcement Learning

RL: environment transition probabilities reward function action next state, immediate reward ENVIROMMENT state

RL: Agent Knowledge: Q(s,a) exploratory Learning: update Q-values state Knowledge representation (FA) Backup table Neural networkNeural... Watkins QL Sarsa ( )... reward action Learning method Behavior

QL: parameters  θ: number of hidden neurons.  T: number of iterations of the learning process.   0 : initial value of the learning rate.   0 : initial value of the exploration rate.  Learning-rate decreasing function.  Exploration-rate decreasing function.

QL: algorithm exploration vs/ exploitation Learning and exploration rates θ (T, T

QL: tuning parameters (observed regularities) 1.(θ,  )-scheme: T=  10 4,  0 = 10 -3,  0 =1, T  =  10 3, T  =v10 3, v  [1,..  ]. PR(θ,  ): best performance with (θ,  )-scheme 2.PR(θ,  ) monotically increase respect  until certain value  (θ) 3.PR(θ,  ) monotically increase respect θ until certain value θ(  ) 4.  (θ) and θ(  ) depend on the problem instance

QL: tuning parameters (methodology: learning schedule given PR Heu ) 1.∆θ =50, θ =0, PR θ =0,  =0, v best =1, 2.While PR θ <PR heu or no-stop 1.θ = θ + ∆θ, PR best =0 2.While PR best ≥ PR  1.  =  +1, T=  10 4, T  =  PR  =PR best, 3.For v= v best to  T  =v10 3 PR[v]=Q-Learning(T, θ, 10 -3, 1, T , T  ) 4.[PR best,v best ]=max(PR) 3.PR θ = PR 

Discussion  For given capacities solve the SMDP with QL  Model other LTC complexities: different facilities and room accommodations, client choice and level of care

Thank you for your attention  Questions?

Neural Network for Q(s,a)