1. Dynamic Treatment Regimes, STAR*D & Voting D. Lizotte, E. Laber & S. Murphy ENAR March 2009

2. 2 Dynamic treatment regimes are individually tailored treatments, with treatment type and dosage changing according to patient outcomes. Operationalize clinical practice. k Stages for one individual Observation available at j th stage Action at j th stage (usually a treatment)

3. 3 Goal : Construct decision rules that input information available at each stage and output a recommended decision; these decision rules should lead to a maximal mean Y. Y is a known function of The dynamic treatment regime is the sequence of two decision rules: k=2 Stages

4. 4 Optimal Dynamic Treatment Regime satisfies

5. 5 Data for Constructing the Dynamic Treatment Regime: Subject data from sequential, multiple assignment, randomized trials. At each stage subjects are randomized among alternative options. A j is a randomized treatment with known randomization probability.

6. 6 STAR*D

7. 7 STAR*D Analyses X 1 includes site, preference for future treatment and can include other baseline variables. X 2 can include measures of symptoms (Qids), side effects, preference for future treatment Y is (reverse-coded) the minimum of the time to remission and 30 weeks.

8. 8 Regression-based methods for constructing decision rules Q-Learning (Watkins, 1989) (a popular method from computer science) Optimal nested structural mean model (Murphy, 2003; Robins, 2004) The first method is equivalent to an inefficient version of the second method, when using linear models, each stages’ covariates include the prior stages’ covariates and the actions are centered to have conditional mean zero.

9. 9 There is a regression for each stage. A Simple Version of Q-Learning – Stage 2 regression: Regress Y on to obtain Stage 1 regression: Regress on to obtain

10. 10 for patients entering stage 2: is the average outcome conditional on patient history (no remission in stage 1; includes past treatment and variables affected by stage 1 treatment). is the estimated average outcome assuming the “best” treatment is provided at stage 2 (note max in formula). is the dependent variable in the stage 1 regression for patients moving to stage 2

11. 11 A Simple Version of Q-Learning – Stage 2 regression, (using Y as dependent variable) yields Stage 1 regression, (using as dependent variable) yields

12. 12 Decision Rules:

13. 13 Measures of Confidence Classical –Confidence/Credible intervals and/or p- values concerning the β 1, β 2. –Confidence/Credible intervals concerning the average response if is used in future to select the treatments.

14. 14 A Measure of Confidence for use in Exploratory Data Analysis Voting –Estimate the chance that a future trial would find a particular stage j treatment best for a given s j. The vote for treatment a j * is

15. 15 A Measure of Confidence for use in Exploratory Data Analysis Voting –If stage j treatment a j is binary, coded in {-1,1}, then

16. 16 The Vote: Intuition If has a normal distribution with variance matrix then is

17. 17 Bootstrap Voting An inconsistent bootstrap vote estimator of is

18. 18 Bootstrap Voting A consistent bootstrap vote estimator of is where is smooth and

19. 19 What does the vote mean? is similar to 1- pvalue for the hypothesis in that it converges, as n increases, to 1 or 0 depending on the sign of If then the limiting distribution is not uniform; instead converges to a constant.

20. 20 STAR*D Regression formula at stage 2:

21. 21

22. 22 STAR*D Regression formula at stage 1:

23. 23 STAR*D Decision Rule for subjects preferring a switch at stage 1 if offer VEN if offer SER if offer BUP

24. 24 STAR*D Level 2, Switch

25. 25

26. 26

27. 27 Truth in Advertising: STAR*D Missing Data + Study Drop-Out 1200 subjects begin level 2 (e.g. stage 1) 42% study dropout during level 2 62% study dropout by 30 weeks. Approximately 13% item missingness for important variables observed after the start of the study but prior to dropout.

28. 28 Truth in Advertising: STAR*D Multiple Imputation within Bootstrap 1000 bootstrap samples of the 1200 subjects Using the location-scale model we formed 25 imputations per bootstrap sample. The stage j Q-function (regression function) for a bootstrap sample is the average of the 25 Q-functions over the 25 imputations.

29. 29 Discussion We consider the use of voting to provide a measure of confidence in exploratory data analyses. Our method of adapting the bootstrap voting requires a tuning parameter. It is unclear how to best select this tuning parameter. We ignored the bias in estimators of stage 1 parameters due to the fact that these parameters are non-regular. The voting method should be combined with bias reduction methods.

30. 30 This seminar can be found at: http://www.stat.lsa.umich.edu/~samurphy/ seminars/ENAR2009.ppt Email me with questions or if you would like a copy! samurphy@umich.edu