1. Dynamic Treatment Regimes: Challenges in Data Analysis S.A. Murphy Survey Research Center January, 2009

2. 2 Outline What are Dynamic Treatment Regimes? Myopic Decision Making Constructing Regimes Q-Learning Example using CATIE

3. 3 Dynamic Treatment Regimes operationalize multi-stage decision making. These are individually tailored sequences of interventions, with intervention type and dosage adapted to the individual. Generalization from a one-time decision to a sequence of decisions concerning interventions Operationalize clinical practice. Each decision corresponds to a stage of intervention

4. 4 Dynamic Treatment Regime “Jobs First” Welfare Program At each stage of intervention –Use individual characteristics (assets, income, age, health, employment), characteristics of the environment (domestic violence, incapacitated family member, # children, living arrangements…), –To select actions/interventions such as child care, job search skills training, amount of cash benefit, medical assistance, education, –In order to maximize long term rewards (maximize employment/independence over longer term).

5. 5

6. 6 Why use a Dynamic Treatment Regime? –High heterogeneity in response to any one intervention What works for one person may not work for another What works now for a person may not work later –Improvement often marred by relapse Remitted or few current symptoms is not the same as cured. –Co-occurring disorders/adherence problems are common

7. 7 Outline What are Dynamic Treatment Regimes? Myopic Decision Making Constructing Regimes Q-Learning Example using CATIE

8. 8 Myopic Decision Making In myopic decision making, decision makers use regimes that seek to maximize immediate rewards. Problems: –Ignore longer term consequences of present actions. –Ignore the range of feasible future actions/interventions –Ignore the fact that immediate responses to present actions may yield information that pinpoints best future actions (A dynamic treatment regime tells us how to use the observations to choose the actions/interventions.)

9. 9 Treatment of Schizophrenia Myopic action: Offer patients a treatment that reduces schizophrenia symptoms for as many people as possible. The result: Some patients are not helped and/or experience abnormal movements of the voluntary muscles (TDs). The class of subsequent medications is greatly reduced. The mistake: We should have taken into account the variety of treatments available to those for whom the first treatment is ineffective. The message: Use an initial medication that may not have as large a success rate but that will be less likely to cause TDs.

10. 10 Treatment of Opioid Dependence Myopic action: Choose an intensive multi-component treatment (methadone + counseling + behavioral contingencies) that immediately reduces opioid use for as many people as possible. The result: Behavioral contingencies are burdensome/expensive to implement and many people may not need the contingencies to improve. The mistake: We should allow the patient to exhibit poor adherence prior to implementing the behavioral contingencies. The message: Use an initial treatment that may not have as large an immediate success rate but carefully monitor patient adherence to ascertain if behavioral contingencies are required.

11. 11 Outline What are Dynamic Treatment Regimes? Myopic Decision Making Constructing Regimes Q-Learning Example using CATIE

12. 12 Basic Idea for Constructing a Regime: Move Backwards Through Stages. (Pretend you are “All-Knowing”)

13. 13 2 Stages for each individual Observations available at j th stage Action at j th stage

14. 14 2 Stages History available at each stage Primary Outcome/Reward:

15. 15 A dynamic treatment regime is the sequence of decision rules: A simple decision rule is: given weights β, switch treatment at stage j if otherwise maintain on current treatment; S j is a vector summary of the history, H j.

16. 16 Goal: Use data to construct decision rules that input information in the history at each stage and output a recommended decision; these decision rules should lead to a maximal mean Y. In the future we employ the actions recommended by the decision rules:

17. 17 Example of Decision Rules Treatment of depression. Goal is to achieve and maintain remission. Provide Citalopram for up to 12 weeks gradually increasing dose as required. If either the maximum dose has been provided for two weeks, or 12 weeks have occurred, yet there is no remission, then if there has been a 50% improvement in symptoms, augment with Mirtazapine. else switch treatment to Bupropion. Else (remission is achieved) maintain on Citalopram and provide web-based disease management.

18. 18 Idealized Data for Constructing the Dynamic Treatment Regime: Data from sequential, multiple assignment, randomized trials in which at each stage subjects are randomized among alternative options. That is, A j is a randomized action with known randomization probability. Binary actions with P[A j =1]=P[A j =-1]=.5

19. 19 Outline What are Dynamic Treatment Regimes? Myopic Decision Making Constructing Regimes Q-Learning Example using CATIE

20. 20 Regression-based methods for constructing decision rules Q-Learning (Watkins, 1989) (a popular method from computer science) A-Learning or optimal nested structural mean model (Murphy, 2003; Robins, 2004) The first method is an inefficient version of the second method when each stages’ covariates include the prior stages’ covariates and the actions are centered to have conditional mean zero.

21. 21 Basic Idea for Constructing a Regime: Move Backwards Through Stages. (Pretend you are “All-Knowing”)

22. 22 (k=2) Dynamic Programming

23. 23 Approximate for S', S vector summaries of the history, A Simple Version of Q-Learning –binary actions Stage 2 regression: Use least squares with outcome, Y, and covariates to obtain Set Stage 1 regression: Use least squares with outcome, and covariates to obtain

24. 24 Approximate for S', S vector summaries of the history, A Simple Version of Q-Learning –binary actions Stage j decision rule: Select treatment = 1 if Otherwise select treatment = -1

25. 25 Outline What are Dynamic Treatment Regimes? Myopic Decision Making Constructing Regimes Q-Learning Example using CATIE

26. 26 Clinical Antipsychotic Trials of Intervention Effectiveness (Schizophrenia) Multi-stage trial of 18 months duration Relaxed entry criteria A large number of sites representing a broad array of clinical settings (state mental health, academic, Veterans’ Affairs, HMOs, managed care) Approximately 1500 patients

27. 27 CATIE Randomizations (simplified) Phase 1 Randomized Treatments OLAN QUET RISP ZIPR PERP Phase 2 Treatment preference Efficacy Tolerability Randomized Treatments CLOZ OLAN QUET RISP OLAN QUET RISP ZIPR Phase 3 Treatments selected many options by preference

28. 28 Preliminary Analyses Reward: Time to Treatment Dropout Phase 1 analysis: –Controls: TD, recent exacerbation, site –Tailoring variable: pretreatment PANSS Phase 2 analysis: –Controls: TD, recent exacerbation, site –Tailoring variables: “treatment preference,” phase 1 treatment, end of phase 1 PANSS Constructing Dynamic Treatment Regimes using CATIE

29. 29

30. 30

31. 31 Preliminary Analyses Myopic versus Non-myopic Analyses Reward: Integrated Quality of Life (QoL) Phase 1 analysis: –Controls: TD, recent exacerbation, site –Tailoring variable: pretreatment QoL Phase 2 analysis: –Controls: TD, recent exacerbation, site –Tailoring variables: “treatment preference,” phase 1 treatment, end of phase 1 QoL

32. 32

33. 33

34. 34 Challenges It is extremely challenging to provide measures of confidence that possess “good frequentist properties.” Clinical Decision Support Systems –We need to be able construct dynamic treatment regimes that recommend a group of treatment actions when there is no evidence that a particular treatment action is best. Even in this randomized trial setting, the most straightforward analyses are subject to confounding bias. Some methods to avoid confounding bias are available.

35. 35 Acknowledgements: This presentation is based on work with many individuals including Eric Laber, Dan Lizotte, John Rush, Scott Stoup, Joelle Pineau, Daniel Almirall and Bibhas Chakraborty,. Email address: samurphy@umich.edu Slides with notes at: http://www.stat.lsa.umich.edu/~samurphy/ Click on seminars > health science seminars

36. 36 Causal Inference Challenges Behavioral/Social/Medical Sciences Incomplete mechanistic models –Unknown causes Use data on individuals to combat the dearth of mechanistic models. –Drawback: non-causal “associations” occur due to the unknown causes of the observations.

37. 37 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

38. 38 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

39. 39 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

40. 40 The problem: Even when treatments are randomized, non-causal associations occur in the data. The solution: Statistical methods should appropriately “average” over the non-causal associations between treatment and reward. Unknown, Unobserved Causes (Incomplete Mechanistic Models)

41. 41 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

42. 42 Unknown, Unobserved Causes Problem: We recruit students via flyers posted in dormitories. Associations between observations and rewards are highly likely to be (due to the unknown causes) non- representative. Solution: Sample a representative group of college students.

43. 43 Summary of Solutions To Causal Problems If possible randomize treatments (e.g. actions). Develop methods that avoid being influenced by non-causal associations yet help you construct the policy. Subjects in your data should be representative of population of subjects.