1. Introduction to Health Systems Engineering
University of Wisconsin - Madison
Fall 2014
ISyE 417
Introduction to Health Systems Engineering
Instructor: Prof. Jingshan Li
Dept. of Industrial and Systems Engineering
University of Wisconsin - Madison

2. Markov Chain

3. Information
Office hour: Tuesday/Thursday 10:00AM-11:00AM, or by appointment
Phone: (608)
Lecture: Tuesday 11:00-12:15, ME 1164
Case study: Thursday 11:00-12:15, Wendt 410A

4. Outline Markov Chain Simulation Lean/Six Sigma
Estimating demand
Planning staffing level
Patient flow modeling
Patient status modeling
Simulation
More
Lean/Six Sigma
Process improvement
Comparison: Manufacturing vs. Healthcare

5. Outline For lectures: For case studies: Exam: November 20, 2014
Preview of lecture notes to prepare for Tuesday class
Continue discussion on Thursday class
Finish online quiz after class, which is due Tuesday the following week
For case studies:
Preview of case study papers to select one for each group
Discuss case studies on Thursday
Write case study report, which is due Thursday the following week
Exam: November 20, 2014

6. Markov Chain (cont.)

7. Issues and Challenges Issues or challenges in healthcare Cost
Accessibility
Quality
Safety
Outcome
Others

8. Issues and Challenges Specific Questions Markov chain models Demand
Scheduling
Staffing
Markov chain models

9. Markov Chain
A stochastic process that takes on a finite or accountable number of possible values
If Xn=i, then the process is said to be in state i at time n
The following condition is satisfied: for all n ≥ 0
The conditional distribution of any future state Xn+1 given the past states X0, X1, …, Xn-1, and the present state Xn, is independent of the past states and only depends on the present state.
pij is the probability that the process will next be in state j when it is in state i

10. One-step Transition Probability Matrix (TPM)
One-step transition matrix

11. Ex: Clinical Trials
Two drugs are available to treat a particular disease, and we need to determine which is more effective. This is generally accomplished by conducting clinical trials.
Suppose drug i is effective with probability pi, i=1,2.
Play the winner rule:
If the n-th patient is given drug i, i=1,2, and it is effective, then the same drug is given to the n+1-th patient; if ineffective, the n+1-th patient is given the other drug. Stick with a drug as long as its results are good; otherwise switch to the other drug.

12. Ex: Clinical Trials Let Xn be the drug given to the n-th patient
Xn=i, drug i
Patients are chosen randomly

13. n-step Transition Probability Matrix (TPM)
Define the n-step transition probability pij(n)
Probability that starting in state i the process will go to state j in n transitions.
The n-step transition matrix can be obtained by multiplying the matrix P by itself n times

14. Limiting Probability (Steady-state Probability)
πj is the unique non-negative solution
It also represents the long-run proportion of time that the process will be in state j
Independent of initial value

15. Markov Chain (cont.)

16. How do we estimate the future demand?

17. Demand for Health Services
How much demand do we have, i.e., how many people will need the health services provided?
How much will that demand change over time?
Can we use information about those changes to plan future services?

18. Forecasting Future Populations
2011
2012
2013
2014
Under 18
2233
2443
2550 ?
18-60
12,344
15,000
17,009
60 & over
8922
8899
9100

19. Forecasting Future Populations
2011
2012
2013
2014
Under 18
2233
2443
2550 ?
18-60
12,344
15,000
17,009
60 & over
8922
8899
9100
210 more people in 2012
5.5% age into the next group
(123)
1% die or leave (22)
355 new children born

20. Forecasting Demand for Services
Using population trends to estimate demand
Population & Service demand data collected by government groups
Regression models of growth over time
Capitalizes on long-run trends
Moving averages and exponential smoothing
Weighs recent experience more than long-run patterns
Markovian Forecasts (population dynamics)
Useful when the estimate of time Ti+1 depends on time Ti

21. Markovian Prediction Assume that population of interest is grouped
Age groups (< 18, 18-45, 46-60, >60)
Severity of illness (mild, moderate, severe)
Members of a group may remain in the group or may leave
Age groups: maturation, death, migration
Severity of illness: improvement or decline
We can estimate the probability of changing states by studying the population dynamics
A Transition Matrix summarizes the movement in and out of each group

22. <18 Mild
<18 Moderate
<18 Severe
18-45 Mild
18-45 Moderate
Severe
46-60 Mild
46-60 Moderate
Severe
>60 Mild
>60 Moderate
>60 Severe
<18
Mild
p1,1
p1,2
p1,3
p1,4
p1,5
p1,6
p1,7
p1,8
p1,9
p1,10
p1,11
p1,12
p2,1
p2,2
p2,3
p2,4
p2,5
p2,6
p2,7
p2,8
p2,9
p2,10
p2,11
p2,12
p3,1
p3,2
p3,3
p3,4
p3,5
p3,6
p3,7
p3,8
p3,9
p3,10
p3,11
p3,12
p4,1
p4,2
p4,3
p4,4
p4,5
p4,6
p4,7
p4,8
p4,9
p4,10
p4,11
p4,12
p5,1
p5,2
p5,3
p5,4
p5,5
p5,6
p5,7
p5,8
p5,9
p5,10
p5,11
p5,12
p6,1
p6,2
p6,3
p6,4
p6,5
p6,6
p6,7
p6,8
p6,9
p6,10
p6,11
p6,12
p7,1
p7,2
p7,3
p7,4
p7,5
p7,6
p7,7
p7,8
p7,9
p7,10
p7,11
p7,12
p8,1
p8,2
p8,3
p8,4
p8,5
p8,6
p8,7
p8,8
p8,9
p8,10
p8,11
p8,12
46-60 Severe
p9,1
p9,2
p9,3
p9,4
p9,5
p9,6
p9,7
p9,8
p9,9
p9,10
p9,11
p9,12
>60
p10,1
p10,2
p10,3
p10,4
p10,5
p10,6
p10,7
p10,8
p10,9
p10,10
p10,11
p10,12
>60 Moderate
p11,1
p11,2
p11,3
p11,4
p11,5
p11,6
p11,7
p11,8
p11,9
p11,10
p11,11
p11,12
p12,1
p12,2
p12,3
p12,4
p12,5
p12,6
p12,7
p12,8
p12,9
p12,10
p12,11
p12,12

23. <18 Mild
<18 Moderate
<18 Severe
18-45 Mild
18-45 Moderate
Severe
46-60 Mild
46-60 Moderate
Severe
>60 Mild
>60 Moderate
>60 Severe
<18
Mild
p1,1
p1,2
p1,3
p1,4
p1,5
p1,6
p2,1
p2,2
p2,3
p2,4
p2,5
p2,6
p3,1
p3,2
p3,3
p3,4
p3,5
p3,6
p4,4
p4,5
p4,6
p4,7
p4,8
p4,9
p5,4
p5,5
p5,6
p5,7
p5,8
p5,9
p6,4
p6,5
p6,6
p6,7
p6,8
p6,9
p7,7
p7,8
p7,9
p7,10
p7,11
p7,12
p8,7
p8,8
p8,9
p8,10
p8,11
p8,12
46-60 Severe
p9,7
p9,8
p9,9
p9,10
p9,11
p9,12
>60
p10,10
p10,11
p10,12
>60 Moderate
p11,10
p11,11
p11,12
p12,10
p12,11
p12,12

24. Markov Chain (cont.)

25. How do we staff the hospital for the next shift?

26. Nursing Resources Nurses work 8 hour shifts Of each shift
about 33% time is in direct care
33% time is in ‘indirect’ care
33% is personal time, communication & process
Thus, each nursing shift yields about 5.5 hours of nursing time

27. What Is Known Nursing Work Process Standards that Patient load
Level I patients need 2 hours of nursing care
Level II patients need 4 hours of nursing care
Level III patients need 7 hours of nursing care
Admissions need 3 hours of nursing care
Discharges need 1 hour of nursing care
Patient load
8 patients awaiting admission
16 patients needing low level of care (level I)
22 patients at the mid-level (level II)
7 patients at high level (level III)
2 patients awaiting discharge

28. Probability of moving from one level of care to anotherII
III D
.60
.30
.10
.80
.07
.03
.20
.70
.05
.25 To

29. Forecast: Multiply the Vector of Current State (C) by the Transition Matrix (T)8 16 22 7 2
.60
.30
.10
.80
.07
.03
.20
.70
.05
.25 ×

30. Multiply the number in each group by the probability of remaining in, or transitioning out of, that group
22.4
20.7
6.6
3.4
(0)(8) +
(0)(16) +
(0)(22) +
(0)(7) +
(0)(0)
(.60)(8) +
(.80)(16) +
(.20)(22) +
(.05)(7) +
(.30)(8) +
(.07)(16) +
(.70)(22) +
(.25)(7) +
(.10)(8) +
(.03)(16) +
(.05)(22) +
(.60)(7) +
(0)(8) +
(.10)(16) +
(.10)(7) + =

31. So the number of patients in each class for the next shift is
Interpreting the numbers (rounding)
Incorporating historical information
Anticipated admissions
Forecasting Staff needs….
Admissions
Level I
22.4
Level II
20.7
Level III
6.6
Discharges
3.4

32. How many Nursing Hours are needed?
Expected Patients
Nursing Demand
Admissions
Level I
22.4
Level II
20.7
Level III
6.6
Discharges
3.4
3 hours
2 hours 44
4 hours 84
7 hours 49
1 hour 3
180 Hours of Nursing Care Needed

33. How many nurses do we need?
180 hours of nursing time needed
5.5 hours provided by each nurse
Thus nurses are needed for the anticipated 52 patients

34. Estimating demand for subsequent periods
Create c^ , a new vector based on
Previously computed vector
Refined estimates of Admissions based on historical data (e.g. an average of 2 admissions per shift)
Admissions
Level I
22.4
Level II
20.7
Level III
6.6
Discharges
3.4

35. Forecast: Multiply the Vector of Current State (C) by the Transition Matrix (T)
22.4
20.7
6.6
3.4
.60
.30
.10
.80
.07
.03
.20
.70
.05
.25 ×

36. Markov Chain (cont.)

37. Work Flow
Nurse
Doctor

38. Work Flow
States:
S(t)={p1(t), m1(t), m2(t)}
p1(t): the number of patients in buffer b1 and its downstream service s2.
mi(t): the status of service si, where mi(t)=1 represents that the service is up (i.e., the doctor or nurse is available to carry out the service) and 0 is down (i.e., the doctor or nurse is not available).
Available/unavailable: exponential with rates λi, μi.
Service rate: ci.
One patient per room.

40. Patient length of stay:

41. Markov Chain (cont.)

42. Patient Rescue

43. Patient Rescue
State 1 “Floor NR”: Patient is in non-risk condition, waiting to be checked by the RN.
State 2 “Nurse NR”: Patient is in non-risk condition and the RN is checking him/her.
State 3 “Floor R”: Patient is in risk condition, waiting to be checked by the RN.
State 4 “Nurse R”: Patient is in risk condition and under RN intervention.
State 5 “MD Int”: Patient is in deteriorating condition and under MD intervention.
State 6 “RRT Int”: Patient is in deteriorating condition and under RRT intervention.
State 7 “ICU”: Patient is in deteriorating condition
µi and λi are transition rates.

44. Patient Rescue

45. Patient Rescue

46. Patient Rescue

47. Markov Chain (cont.)

48. Discussion Questions
What assumptions were needed to make such forecast and analysis?
How can we confirm those assumptions?
What will happen if these assumptions are not valid?
How can we obtain or collect the data?
What is the effect of data accuracy?
What will happen if system dimension increases?

49. Discussion Questions How can we validate the model results?
What can we do by using the model?
What can we learn from such dynamics models that we can’t learn from examining trends in the data bases?
Remember -- these are planning tools