1. Statistical Issues in Developing Adaptive Treatment Strategies for Chronic Disorders S.A. Murphy Univ. of Michigan CDC/ATSDR: March, 2005

2. Setting: Management of chronic, relapsing disorders such as alcohol, cocaine addiction, ADHD, and mental illness Characteristics: May need a sequence of treatments prior to improvement Improvement marred by relapse Intervals during which more intense treatment is required alternate with intervals in which less treatment is sufficient

3. Adaptive Treatment Strategies are individually tailored treatments, with treatment type and dosage changing with ongoing subject information. Mimic Clinical Practice. Brooner et al. (2002) Treatment of Opioid Addiction Breslin et al. (1999) Treatment of Alcohol Addiction Prokaska et al. (2001) Treatment of Tobacco Addiction Rush et al. (2003) Treatment of Depression

4. EXAMPLE : Treatment of alcohol dependency. Primary outcome is a summary of heavy drinking scores over time.

6. T Decisions Observations made prior to j th decision Treatment at j th decision Primary Outcome: for a known function f

7. An adaptive treatment strategy is a vector of decision rules, one per decision If the strategy is implemented then

8. Why is it useful to have data in which all treatment sequences are possible in developing adaptive treatment strategies? Sequential Multiple Assignment Randomized Trial.

9. Why not run multiple trials? Why not run two trials and choose the best initial treatment on the basis of the first trial and then choose the best secondary treatment on the basis of a randomized trial of secondary treatments? The comparison of initial treatments that are part of a strategy is different from a comparison of stand-alone treatments. Cohort effects.

12. Evaluating the initial treatments requires that we first calculate the mean of the primary outcome of patients for each combination of (a) secondary treatment, (b) intermediate outcome and (c) initial treatment. Why not run multiple trials? Why not run two trials and combine over the trials using medical decision making methods (Parmigiani, 2002)?

13. Evaluating the initial treatments requires that we first calculate the mean of the primary outcome of patients for each combination of (a) secondary treatment, (b) intermediate outcome and (c) initial treatment. Why not run multiple trials? If given (b) the intermediate outcome, the primary outcome to secondary treatment does not vary by (c) the initial treatment then medical decision making methods can be used.

15. Evaluating the initial treatments requires that we first calculate the mean of the patient outcome to each combination of (a) secondary treatment, (b) intermediate outcome and (c) initial treatment. When might the outcome to secondary treatment vary by (c) initial treatment even after we condition on the intermediate outcome?

18. The Bottom Line Are there unobserved/unknown but potentially important common causes of the primary and intermediate outcomes? Are there important causal pathways from initial treatment to primary outcome; pathways that do not include the intermediate outcome? If yes to either of the above then use sequential multiple assignment randomized trials to develop good adaptive treatment strategies.

19. An Aside!

22. Examples of sequential multiple assignment randomized trials: CATIE (2001) Treatment of Psychosis in Alzheimer’s Patients CATIE (2001) Treatment of Psychosis in Schizophrenia STAR*D (2003) Treatment of Depression Thall et al. (2000) Treatment of Prostate Cancer

23. Statistical Analysis Screen for active components (????; Murphy,2004) Secondary Analyses Compare the mean response to two different simple strategies Estimate the parameters in the best decision rules ----Reinforcement Learning---

25. Reinforcement Learning Indirect Methods Fit a model to the data --enumerate the strategies (Thall, 2000) --sample from the fitted distribution and use (approximate) dynamic programming (Müller et al., 1999; Tsitsiklis & van Roy, 1996; Simester, Sun & Tsitsiklis, 2003) or use temporal-difference methods (Sutton & Barto, 1998)

26. Reinforcement Learning Direct Methods Directly use limited data to estimate parameters in the decision rules. Q-Learning (Watkins, 1989; Sutton & Barto, 1998) A-Learning (Murphy, 2003; Robins, 2003) Direct Search Methods: ????? ; Weighting + enumeration of strategies (Murphy, et al., 2001)

27. Additional Open Problems Dealing with high dimensional X –In this setting all present methods are biased except for a few methods that have high variance! –Feature construction. Constructing the best adaptive treatment strategy when there are constraints on the decision rules. Good experimental designs.

28. This seminar can be found at: http://www.stat.lsa.umich.edu/~samurphy/seminars/ cdc0305.ppt My email address: samurphy@umich.edu