1. Treatment Effect Heterogeneity & Dynamic Treatment Regime Development S.A. Murphy

2. 2 Dynamic treatment regimes (DTRs) are individually tailored treatments, with treatment type and dosage changing according to individual outcomes. ***utilize treatment effect heterogeneity to individualize treatment***

3. 3 Example of a DTR Adaptive Drug Court Program for drug abusing offenders. Goal is to minimize recidivism and drug use. Marlowe et al. (2008, 2009, 2011)

4. 4 Adaptive Drug Court Program

5. Treatment Effect Heterogeneity Focus on Theory: Used to deepen understanding of underlying causal, mechanistic structure Focus on Practice: Used to improve decision making in practice –For Whom, When, and in Which Context, might a specific treatment be most useful? –This is our focus today

6. Treatment Effect Heterogeneity & DTR Development Take Advantage of Treatment Effect Heterogeneity in Design of Intervention Trial –Embedded tailoring variables –Part of “treatment action” Take Advantage of Treatment Effect Heterogeneity in Design of the DTR. –Data analyses

7. Pelham ADHD Study Begin low dose Med 8 weeks Assess- Adequate response? Continue, reassess monthly; randomize if deteriorate Intensify Current Treatment Random assignment: Augment with other Treatment No Begin low-intensity BMOD 8 weeks Assess- Adequate response? Continue, reassess monthly; randomize if deteriorate Augment with other treatment Random assignment: Intensify Current Treatment Yes No Random assignment:

8. Txt Effect Heterogeneity  Embedded Tailoring Variable Embedded Tailoring Variables: (a) Teacher reported Impairment Scale, (b) Teacher reported individualized list of target behaviors Non-response is assessed at 8 weeks and every 4 weeks thereafter.

9. Txt Effect Heterogeneity  Embedded DTRs 4 Embedded DTRs 1)Start with BMOD; only if nonresponse criterion reached, augment with MED 2)Start with BMOD; only if nonresponse criterion reached, intensify BMOD 3)Start with MED; only if nonresponse criterion reached, augment with BMOD 4)Start with MED; only if nonresponse criterion reached, intensify MED

10. Oslin Alcoholism Trial Late Trigger for Nonresponse 8 wks Response TDM + NTX CBI +MM Random assignment: CBI +NTX+MM Nonresponse Early Trigger for Nonresponse Random assignment: NTX 8 wks Response Random assignment: CBI +NTX+MM CBI+MM TDM + NTX NTX Nonresponse

11. Txt Effect Heterogeneity  Embedded Tailoring Variable & Embedded DTR Embedded Tailoring Variable: heavy drinking days (HDD) First randomization is between treatment actions: move to stage 2 if 2 HDDs versus move to stage 2 if 5 HDDs 8 Embedded DTRs

12. 12 A Data Analysis Method for Utilizing Treatment Effect Heterogeneity to Construct a “More Deeply Tailored” DTR: Q-Learning Subject data from sequential, multiple assignment, randomized trials. At each stage subjects are randomized among alternative options. A j is a randomized action with known randomization probability. Binary actions with P[A j =1]=P[A j =-1]=.5

13. 13 Dynamic Treatment Regime (DTR) The DTR is given by a sequence of decision rules, one per stage of treatment (here 2 stages) DTR= Goal : Construct for which the expected outcome is maximal.

14. 14 Q-Learning (Watkins, 1989; Ernst et al., 2005; Murphy, 2005) (a popular method from computer science)—generalizes regression to multiple stages Q-Learning uses dynamic programming arguments combined with linear regression estimation of conditional means. Q-Learnin g

15. 15 There is a regression for each stage. Simple Version of Q-Learning – Stage 2 regression: Regress Y on to obtain Stage 1 regression: Regress on to obtain

16. 16 for subjects entering stage 2: is the predicted end of stage 2 response when the stage 2 treatment is equal to the “best” treatment. is the dependent variable in the stage 1 regression for patients moving to stage 2

17. 17 A Simple Version of Q-Learning – Stage 2 regression, (using Y as dependent variable) yields Arg-max over a 2 yields

18. 18 A Simple Version of Q-Learning – Stage 1 regression, (using as dependent variable) yields Arg-max over a 1 yields

19. Pelham ADHD Study Begin low dose Med 8 weeks Assess- Adequate response? Continue, reassess monthly; randomize if deteriorate Intensify Current Treatment Random assignment: Augment with other Treatment No Begin low-intensity BMOD 8 weeks Assess- Adequate response? Continue, reassess monthly; randomize if deteriorate Augment with other treatment Random assignment: Intensify Current Treatment Yes No Random assignment:

20. 20 (X 1, A 1, R 1, X 2, A 2, Y) –Y = end of year school performance –R 1 =1 if early responder; =0 if early non-responder –X 2 includes the month of non-response, M 2, and a measure of adherence in stage 1 (S 2 ) –S 2 =1 if adherent in stage 1; =0, if non-adherent –X 1 includes baseline school performance, Y 0, whether medicated in prior year (S 1 ), ODD (O 1 ) –S 1 =1 if medicated in prior year; =0, otherwise. ADHD Example

21. 21 Stage 2 regression for Y: Stage 1 outcome: ADHD Example

22. 22 IF medication was not used in the prior year THEN begin with BMOD; ELSE select either BMOD or MED. IF the child is nonresponsive and was non- adherent, THEN augment present treatment; ELSE IF the child is nonresponsive and was adherent, THEN select intensification of current treatment. Dynamic Treatment Regime Proposal

23. 23 High dimensional data; investigators want to collect real time data Feature construction & Feature selection Many stages or infinite horizon This seminar can be found at: http://www.stat.lsa.umich.edu/~samurphy/ seminars/JSM_Txt_Heterogeneity2012.ppt Future Challenges