1. Brennan Spiegel, MD, MSHS
Markov Models II
HS 249T Spring 2008
Brennan Spiegel, MD, MSHS
VA Greater Los Angeles Healthcare System
David Geffen School of Medicine at UCLA
UCLA School of Public Health
CURE Digestive Diseases Research Center
UCLA/VA Center for Outcomes Research and Education (CORE)

2. Topics More on Markov models versus decision trees
More examples of Markov models
Calculating annual transition probabilities
Time independent (Markov chains)
Time dependent (Markov processes)
Temporary and tunnel states
Half-cycle corrections

3. Disadvantages of Traditional Decision Trees
Limited to one-way progression without opportunity to “go back”
Can become unwieldy in short order
Difficult to capture the dynamic path of moving between health states over time
Often fails to accurately reflect clinical reality

4. Markov Models Allow dynamic movement between relevant health states
Allow enhanced flexibility to better emulate clinical reality
Acknowledge that different people follow different paths through health and disease

5. Inadomi et al. Ann Int Med 2003
Example Markov Model
Important points:
-- Absorbing states
-- Not linear or serial
-- Allowed to go “backwards”
Most critical part of CEA. Musts agree with basic model first, then examine the transition probabilities.
Inadomi et al. Ann Int Med 2003

6. Markov Model
Alive
No Barrett
Barrett
Dead
Year 0

7. Markov Model Alive Barrett Alive No Barrett Dead No Barrett Dead

8. Markov Model Alive Barrett Alive No Barrett Dead No Barrett Dead

9. Markov Model Year 1 Alive Barrett Alive No Barrett Dead No Barrett

10. Markov Model Alive Barrett Alive No Barrett Dead No Barrett Dead

11. Markov Model Alive Barrett Alive No Barrett Dead No Barrett Dead

12. Markov Model End Alive Barrett Alive No Barrett Dead No Barrett Dead

13. Decision Trees and Markov Models may Co-Exist
Both provide different types of information
Information from both is not mutually exclusive
Markov model can be “tacked” onto end of a traditional decision tree

14. Chronic HBV No Cirrhosis Normal Lifespan No Therapy Cirrhosis
Markov Model
Normal Lifespan
No Therapy
Virological Response
Normal Lifespan
No Response
No Cirrhosis
Cirrhosis
Markov Model
Inteferon
Chronic
HBV
Lamivudine
Adefovir
Adefovir Salvage
Response
No Response
Start Adefovir
No Resistance
Resistance
Con’t Lamivudine

15. To Cirrhosis Markov Model
Uncomplicated
Cirrhosis
Chronic HBV
Virological Resistance
Chronic HBV on Treatment
Virological Response
Virological Relapse

16. Markov Model #2 Uncomplicated Cirrhosis Complicated Hepatocellular
Carcinoma
Liver
Transplant
Death

17. No GI or CV Complications
Dyspepsia
Myocardial Infarction
GI Bleed
Post Myocardial Infarction
Post
GI Bleed
Death

18. Hepatocellular Cancer Liver Transplantation
START
Sub-Clinical HE
Overt HE
Clinical Response
Hepatocellular Cancer
Non-HE Complication
Liver Transplantation
Death

19. Annual Probability Estimates
Cirrhosis in HBeAg(-) % Cirrhosis in HBeAg(+) % Chronic HBV  liver cancer % Cirrhosis  liver cancer % Compensated cirrhosis  decompensated % Decompensated cirrhosis  liver transplant % Liver cancer  liver transplant % Death in compensated cirrhosis % Death in decompensated cirrhosis % Death in liver cancer %

20. Converting Data Into Annual Probability Estimates
Cannot simply divide long-term data by number of years
Example:
If 5-year risk of an event is 40%, then annual risk does not amount to: 40 5 = 8%

21. Converting Data Into Annual Probability Estimates
General rule for converting long-term data into annual probabilities:
1-(1-x)Y = Probability at Y Years

22. Example of Converting Long Term Data into Annual Probability
If probability of bleed at 5 years = 0.40, then the annual probability = x, as follows: 
1- (1-x)5 = 0.40 
(1-x)5 = 1 – 0.40 
(1-x)5 = 0.60 
(1-x) = 0.902
x = 0.097
… or 9.7%

23. Example of Converting Long Term Data into Annual Probability
Check for errors by back calculating using the inverse equation:
1-(1-annual probability)Y = probability at Y years 
1-( )5 = 0.40 
1-(0.903)5 = 0.40
0.40 = 0.40

24. Markov Cycle Converter
Forward Calculator
Enter Percentage to be Converted 40
Enter Number of Cycles 5
Cycle Probability=
Backwards Calculator
Enter Cycle Probability for Conversion
0.097
Converted Probability=
Converted Percentage=

25. Steps to Combining Time-Independent Transition Probabilities
Step 1  Collect and abstract relevant studies
Step 2  Select common cycle length
Step 3  Convert all studies to common cycle length units
Step 4  Calculate common cycle transition probabilities
Step 5  Combine common cycle probabilities

26. Calculated 12-Month Probability
Example
Study
Study Duration
Number of 12 Mo Cycles
End Percentage
Calculated 12-Month Probability
Jones
6 months
0.5
12%
0.23
James
12 months 1
19%
0.19
Johnson
18 months
1.5
22%
0.15
Marshall
3 months
0.25 8%
0.28
Mean = 21.3% / 12-month cycle

27. Many Probabilities are Time Dependent
Time independence is usually a simplifying assumption
Progress though many systems in health care (biological, organizational, psychosocial, etc) are erratic and non-linear
May need to account for time-dependent transitional probabilities using:
Tables
Tunnels

28. Using Tables for Time-Dependent Probabilities
Tables allow transition probabilities to vary cycle- by-cycle
Allow greater precision for processes that are non- linear
0.05 8 7 6 5 4 3 2 1
Probability
Cycle
0.02
0.03
0.04
0.06
0.07
0.08
0.1
Time Independent
Time Dependent

29. Probability Cycle Time Dependent: Non-linear Accelerating Returns
Time Independent: Linear Curve
Time Dependent: Non-linear Diminishing Returns
Probability
Cycle

30. Using Tunnels States
Some events can interfere with otherwise orderly Markov chains
Can get “stuck in a rut” that removes subjects from the usual flow of events
e.g. developing cancer
Tunnel states add flexibility to Markov models:
Model getting “stuck in the rut”
Compartmentalize processes into component states
Can model various “recovery states” from the “rut”
Can incorporate time-dependent transitions

31. Example Prior to Tunnel State

32. Example With Tunnel State

33. Half-Cycle Corrections
In “real life,” events can occur anytime during a given cycle – it is usually a random event
The default setting for Markov models is for events to occur at the exact end of each cycle
Yet the default setting can lead to errors in the calculation of average values
Will tend to overestimate benefits (e.g. life expectancy) by about half of a cycle

34. Rationale for Half-Cycle Corrections
“In whatever cycle a ‘member’ of the cohort analysis dies, they have already received a full cycle’s worth of state reward, at the beginning of the cycle. In reality, however, deaths will occur halfway through a cycle on average. So, someone that dies during a cycle should lose half of the reward they received at the beginning of the cycle.”
- TreeAge Pro Manual, p476

35. 1.0
0.8
0.6
0.4
0.2
0.0
Proportion Alive
AUC=2
Cycle

36. 1.0
0.8
0.6
0.4
0.2
0.0
Proportion Alive
AUC=2.5
Cycle

37. 1.0
0.8
0.6
0.4
0.2
0.0
Proportion Alive
AUC=2.0ish
Cycle