1. Saturday Agenda Interfaces – Johnson, Robert R. How Bayer Makes Decisions to Develop New Drugs – Isbrandt, Derek A Major League Baseball Team Uses Operations Research to Improve Draft Preparation – Agnew, Joseph G. Kaizen and Stochastic Networks Support the Investigation of Aircraft Failures – Tran, Binh Patient-Centered Care: A Simulation Model to Compare Strategies forthe Reduction of Health-Care- Associated Markov Chains Networks Big Data and Analytics (Guest lecturers) STAT 5802 – Markov Chains

2. Markov Chains Andrey Markov 1856-1922

3. Markov Chains General Description We want to describe the behavior of a system as it moves (makes transitions) probabilistically from “state” to “state”. States may be qualitative or quantitative Basic Assumption – The future depends only on the present (current state) and not on the past. That is, the future depends on the state we are in, not on how we arrived at this state.

4. Example 1 - Brand loyalty or Market Share For ease, assume that all cola buyers purchase either Coke or Pepsi in any given week. That is, there is a duopoly. Assume that if a customer purchases Coke in one week there is a 90% chance that the customer will purchase Coke the next week (and a 10% chance that the customer will purchase Pepsi). Similarly, 80% of Pepsi drinkers will repeat the purchase from week to week.

5. Example 1 - Developing the Markov Matrix States – State 1 - Coke was purchased – State 2 - Pepsi was purchased – (note: states are qualitative) Markov (transition or probability) Matrix From\ToCokePepsi Coke0.90.1 Pepsi0.20.8

6. Example 1 – Understanding Movement From\ToCokePepsi Coke0.90.1 Pepsi0.20.8 Quiz: If we start with 100 Coke purchasers and 100 Pepsi purchasers, how many Coke purchasers will there be after 1 week?

7. Graphical Description – 1 The States From\To CokePepsi Coke.9.1 Pepsi.2.8

8. Graphical Description – 2 Transitions from Coke From\To CokePepsi Coke.9.1 Pepsi.2.8.9.1

9. Graphical Description – 3 All transitions From\To CokePepsi Coke.9.1 Pepsi.2.8.9.1.8.2

10. Example 1 - Starting Conditions Percentages – Identify probability of (percentage of shoppers) starting in either state (We will assume a 50/50 starting market share in our example that follows.) – Assume we start in one specific state (by setting one probability to 1 and the remaining probabilities to 0) Counts (numbers) – Identify number of shoppers starting in either state

11. Example 1 From\To Coke Pepsi Coke0.90.1 Pepsi0.20.8 Starting Probabilities = 50% (or 50 people) each Questions – What will happen in the short run (next 3 periods)? – What will happen in the long run? – Do starting probabilities influence long run?

12. Graphical Solution After 1 Transition From\To CokePepsi Coke.9.1 Pepsi.2.8 (50)Coke(55)(50)Pepsi(45).9(50)=45.1(50)=5.8(50)=40.2(50)=10

13. Graphical Solution After 2 Transitions From\To CokePepsi Coke.9.1 Pepsi.2.8 (55)Coke(58.5)(45)Pepsi(41.5).9(55)=49.5.1(55)=5.5.8(45)=36.2(45)=9

14. Graphical Solution After 3 Transitions (58.5)Coke(60.95)(41.5)Pepsi(39.05).9(58.5)=52.65.1(58.5)=5.85.8(41.5)=33.2.2(41.5)=8.3

15. Analyzing Markov Chains Using QM for Windows – Module – Markov Chains Number of states – 2 Number of transitions - 3

16. Example 1 – After 3 transitions n-step Transition probabilities End of Period 1CokePepsi Coke0.89990.1000 Pepsi0.20000.8000 End prob (given initial)0.55000.4500 End of Period 2 CokePepsi Coke0.82990.1700 Pepsi0.34000.6600 End prob (given initial)0.58490.4150 End of Period 3 CokePepsi Coke0.78090.2190 Pepsi0.43800.5620 End prob (given initial)0.60940.3905 1 step transition matrix 2 step transition matrix 3 step transition matrix

17. Example 1 - Results (3 transitions, start =.5,.5) From\To Coke Pepsi Coke0.781000.21900 Pepsi0.438000.56200 Ending probability0.6095 0.3905 Steady State probability0.66660.3333 Note: We end up alternating between Coke and Pepsi 3 step transition matrix Depends on initial conditions Independent of initial conditions

18. Example 2 - Student Progression Through a University States – Freshman – Sophomore – Junior – Senior – Dropout – Graduate – (note: again, states are qualitative)

19. Example 2 - Student Progression Through a University - States FreshmanSophomoreJuniorSenior Drop outGraduate Note that eventually you must end up in Grad or Drop-out.

20. Example 2 – Results Lazarus paper data First yrSophJuniorSeniorGradDrop out First year0.00000.00000.00000.00000.85650.1434 Sophomore0.00000.00000.00000.00000.88600.1139 Junior0.00000.00000.00000.00000.92730.0726 Senior0.00000.00000.00000.00000.96900.0310 Graduate0.00000.00000.00000.00001.00000.0000 Drop out0.00000.00000.00000.00000.00001.0000 End prob00000.85650.1434 Steady State 000011

21. From the paper If there are an equal number of freshmen, sophomores, juniors and seniors at the beginning of an academic year then The percentage of this mixed group of students who will graduate is (.857+.886+.927+.969)/4 = 91%

22. Classification of states Absorbing – Those states such that once you are in you never leave. Graduate, Drop Out Recurrent – Those states to which you will always both leave and return at some time. Coke, Pepsi Transient – States that you will eventually never return to Freshman, Sophomore, Junior, Senior

23. State Classification Exercise State 1State 2State 3 State 4State 5 Absorbing Recurrent Transient

24. State Classification Article “A non-recursive algorithm for classifying the states of a finite Markov chain” European Journal of Operational Research Vol 28, 1987

26. Example 3 - Diseases States – no disease – pre-clinical (no symptoms) – clinical – death – (note: again states are qualitative) Purpose – Transition probabilities can be different for different testing or treatment protocols

27. Example 4 - Customer Bill paying States – State 0: Bill is paid in full – State i: Bill is in arrears for i months, i= 1,2,…,11 – State 12: Deadbeat

28. Example 5 - Oil Market State – State 0 - oil market is normal – State 1 - oil market is mildly disrupted – State 2 - oil market is severely disrupted – State 3 - oil production is essentially shut down – Note: States are qualitative “Tensions around Iran have been supporting the prices in the past years, but the impact might diminish as the United States has said it was ready to release fuel from its Strategic Petroleum Reserve to protect the world economy should oil prices spike.”Iraneconomy http://www.reuters.com/article/2013/01/25/mercuria-oil-idUSL6N0AUBP320130125

29. Example 6 – HIV infections Based on “Can Difficult-to-Reuse Syringes Reduce the Spread of HIV among Injection Drug Users” – Caulkins, et. al. – Interfaces, Vol 28, No. 3, May-June 1998, pp 23-33 State – State 0 – Syringe is uninfected – State 1 – Syringe is infected Notes: – P(0, 1) =.14 14% of drug users are infected with HIV – P(1, 0) =.33+.05 5% of the time the virus dies; 33% of the time it is killed by bleaching

30. Example 7 – Mental Health Lazarus depressed manic euthymic/remitted mortality

31. Example 8 - Baseball States – State 0 - no outs, bases empty – State 1 - no outs, runner on first – State 2 - no outs, runner on second – State 3 - no outs, runner on third – State 4 - no outs, runners on first, second – State 5 - no outs, runners on first, third – State 6 - no outs, runners on second, third – State 7 - no outs, runners on first, second, third – …. Repeat for 1 out and 2 outs for a total of 24 states Moneyball by Michael Lewis, p 134 Moneyball

32. Example 9 – Football Overtime Playoffs (no time limit) States – Team A has ball – Team B has ball – Team A scores (absorbing) – Team B scores (absorbing) “Win, Lose, or Draw: A Markov Chain Analysis of Overtime in the National Football League”, Michael A. Jones, The College Mathematics Journal, Vol. 35, No. 5, November 2004, pp 330-336

33. http://interfaces.journal.informs.org/search?fulltext=markov&submit=yes&x=10&y=11

34. EXERCISE STAT 5802 – Markov Chains

35. Vinayak, Prior, Shinkus, Capozzi, Ltd is considering leasing one of two possible machines – either from Wang, Inc. or from Koger, Inc.. At the beginning of each day, either machine can be found in operating condition or nonoperating condition. The daily transition matrix of the two machines under identical maintenance is given below: For example, if the Wang machine is operating at the beginning of one day there is a 95% chance that it will be operating at the beginning of the next day. Assume that everything (leasing costs, repair costs, etc.) except for the transition matrices are identical for the two machines. Which machine should Capozzi lease and why? WANGOperatingNonopera ting KOGEROperatingNonoperatin g Operating.95.05Operating.98.02 Nonoperating.90.10Nonoperating.85.15

36. STAT 5802 – Markov Chains Assume that at the beginning of work on Monday the Wang machine is in operating condition. The probability that the machine will be in operating condition at the beginning of Friday (4 days later) is _______________. Assume that at the beginning of work on Monday the Wang machine is in operating condition. The probability that the machine will be in operating condition at the beginning of all 4 days from Monday to Friday is __________________. In the context of this problem explain the meaning of a recurrent state. ower right corner) which means that a new machine must be purchased. Write the new transition matrix for Koger. What type of state is the new state? ____________________________ What type of state are operating and nonoperating? ____________________ WANGOperatingNonopera ting KOGEROperatingNonoperatin g Operating.95.05Operating.98.02 Nonoperating.90.10Nonoperating.85.15

37. STAT 5802 – Markov Chains Suppose that the original model should have included the fact that for the Koger model if a machine is inoperable on one day that there is a 10% chance that it may be nonoperating the next day and a 5% chance it may be totally unrepairable (rather than the 15% chance that is listed in the lower right corner) which means that a new machine must be purchased. Write the new transition matrix for Koger. What type of state is the new state? What type of state are operating and nonoperating? WANGOperatingNonopera ting KOGEROperatingNonoperatin g Operating.95.05Operating.98.02 Nonoperating.90.10Nonoperating.85.15

38. Markov Chains The end