1. 1 Possible Roles for Reinforcement Learning in Clinical Research S.A. Murphy November 14, 2007

2. 2 Outline Goal: Improving Clinical Decision Support Systems Using Data –Clinical Decision Support Systems –Critical Decisions –Types of Data –Challenges Incomplete, primitive, mechanistic models Measures of Confidence –Clinical Trials

3. 3

4. 4

5. 5 Questions Patient Evaluation Screen with MSE

6. 6

7. 7 Outline –Clinical Decision Support Systems –Critical Decisions –Types of Data –Challenges Incomplete mechanistic models Measures of Confidence –Clinical Trials

8. 8 Critical Decisions Which treatments should be offered first? How long should we wait for these treatments to work? How long should we wait before offering a transition to a maintenance stage? Which treatments should be offered next?

9. 9 Critical Decisions All of these questions relate to the formulation of a policy. Actions include medications, behavioral therapies, delivery mechanisms, monitoring Observations include biological measures, family history, severity, side effects, functionality, symptoms Rewards include functionality, side effects, symptoms

10. 10 Outline –Clinical Decision Support Systems –Critical Decisions –Types of Data –Challenges Incomplete mechanistic models Measures of Confidence –Clinical Trials

11. 11 Types of Data Large Observational Data Sets –Actions are not manipulated by scientist Clinical Trial Data –Actions are manipulated by scientist Bench research on cells/animals/humans

12. 12 Clinical Trial Data Sets Experimental trial data collected for research purposes –Scientists decide proactively which data to collect and how to collect this data –Use scientific knowledge to enhance the quality of the proxies for observation, reward –Actions are manipulated (randomized) by scientist –Short Horizon (less than 5) –Hundreds of subjects.

13. 13 Observational Data Sets Observational data collected for research purposes –Scientists decide proactively which data to collect and how to collect this data –Use scientific knowledge to enhance the quality of the proxies for observation, action, reward –Actions are not manipulated by scientist –Moderate Horizon –Hundreds to thousands of subjects.

14. 14 Observational Data Sets Clinical databases or registries– (an example in the US would be the VA registries) –Data was not collected for research purposes –Use gross proxies to define observation, action, reward –Moderate to Long Horizon –Thousands to Millions of subjects

15. 15 Outline –Clinical Decision Support Systems –Critical Decisions –Types of Data –Challenges Incomplete mechanistic models Measures of Confidence –Clinical Trials

16. 16 Availability of Mechanistic Models In many areas of RL, scientists can use mechanistic theory, e.g., physical laws, to model or simulate the interrelationships between observations and how the actions might impact the observations. Scientists know many (the most important) of the causes of the observations and know a model for how the observations relate to one another.

17. 17 Incomplete Mechanistic Models in Medical Sciences Scientists who want to use data on individuals to construct policies must confront the fact that non-causal “associations” occur due to the unknown causes of the observations.

18. 18 Conceptual Structure in the Medical Sciences (observational data)

19. 19 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

20. 20 Unknown, Unobserved Causes (Incomplete Mechanistic Models) Problem: Non-causal associations between treatment (here counseling) and rewards are likely. Solutions: –Collect clinical trial data in which treatments are randomized. This breaks the non-causal associations yet permits causal associations. –Prior to applying methods to observational data proactively brainstorm with domain experts to ascertain and measure the main determinants of treatment selection. Then take advantage of causal inference methods designed to minimize assumptions on the data

21. 21 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

22. 22 Unknown, Unobserved Causes (Incomplete Mechanistic Models)

23. 23 The problem: Even when treatments are randomized, non- causal associations occur in the data. Solutions: – Recognize that parts of the Q-function/transition probabilities can not be informed by domain expertise as these parts reflect non- causal associations –Or use methods for constructing policies that “average” over the non-causal associations between action and reward. I think that the importance of this second causal inference problem depends on the kind of data and how you use it. Unknown, Unobserved Causes (Incomplete Mechanistic Models)

24. 24 Measures of Confidence –Measures of confidence are essential Noisy data Need to know when any one of a subset of actions will yield the best rewards –that is, when there is no or little evidence otherwise. It is important to minimize the number of observations that must be collected in the clinical setting

25. 25 Measures of Confidence We would like measures of confidence for the following: –To compare the value of two estimated policies (both estimated using the training data). –To assess if there is sufficient evidence that a particular observation (e.g. output of a biological test) should be part of the policy. –To assess if there is sufficient evidence that a subset of the actions lead to better rewards for a given observation than the remaining actions.

26. 26 Measures of Confidence I must both learn the policy and provide an evaluation of the policy using one data set. The data set is small

27. 27 Measures of Confidence Traditional methods for constructing measures of conference require differentiability (if frequentist properties are desired). Q-functions are constructed via non- differentiable operations (e.g. maximization). The value of a policy is a non-differentiable function of the policy.

28. 28 Outline –Clinical Decision Support Systems –Critical Decisions –Types of Data –Challenges Causal ::: Unknown, unobserved causes Measures of Confidence –Clinical Trials

29. 29 Clinical Trials Data from the --short horizon– clinical trials make excellent test beds for combinations of supervised/unsupervised and reinforcement learning methods. –Developing methods for variable selection in decision making (in addition to variable selection for prediction) –Model selection when goal is learning good policies. –Confidence intervals for the difference in value between two policies. –Feature Construction

30. 30 ExTENd Ongoing study at U. Pennsylvania (D. Oslin) Goal is to learn how best to help alcohol dependent individuals reduce alcohol consumption.

31. 31 Oslin ExTENd Late Trigger for Nonresponse 8 wks Response TDM + Naltrexone CBI Random assignment: CBI +Naltrexone Nonresponse Early Trigger for Nonresponse Random assignment: Naltrexone 8 wks Response Random assignment: CBI +Naltrexone CBI TDM + Naltrexone Naltrexone Nonresponse

32. 32 Adaptive Treatment for ADHD Ongoing study at the State U. of NY at Buffalo (B. Pelham) Goal is to learn how best to help children with ADHD improve functioning at home and school.

33. 33 ADHD Study B. Begin low dose medication 8 weeks Assess- Adequate response? B1. Continue, reassess monthly; randomize if deteriorate B2. Increase dose of medication with monthly changes as needed Random assignment: B3. Add behavioral treatment; medication dose remains stable but intensity of bemod may increase with adaptive modifications based on impairment No A. Begin low-intensity behavior modification 8 weeks Assess- Adequate response? A1. Continue, reassess monthly; randomize if deteriorate A2. Add medication; bemod remains stable but medication dose may vary Random assignment: A3. Increase intensity of bemod with adaptive modifi- cations based on impairment Yes No Random assignment:

34. 34 Studies under review H. Jones study of drug-addicted pregnant women (goal is to reduce cocaine/heroin use during pregnancy and thereby improve neonatal outcomes) J. Sacks study of parolees with substance abuse disorders (goal is reduce recidivism and substance use)

35. 35 Jones’ Study for Drug-Addicted Pregnant Women rRBT 2 wks Response rRBT tRBT Random assignment: rRBT Nonresponse tRBT Random assignment: aRBT 2 wks Response Random assignment: eRBT tRBT rRBT Nonresponse

36. 36 Sack’s Study of Adaptive Transitional Case Management Standard Services Standard TCM Random assignment: 4 wks Response Standard TCM Augmented TCM Standard TCM Nonresponse

37. 37 Discussion Methods for online updating the policy as data accumulates. Methods for producing composite rewards. –High quality elicitation of functionality Human-Computer interface Improving tactics

38. 38 This seminar can be found at: http://www.stat.lsa.umich.edu/~samurphy/ seminars/UAlberta07.ppt Email me with questions or if you would like a copy: samurphy@umich.edu

39. 39 Unknown, Unobserved Causes Problem: We recruit students via flyers posted in dormitories. Associations between observations and rewards are highly likely to be (due to the unknown causes) non- representative. Solution: Sample a representative group of college students.

40. 40 STAR*D This trial is over and the data is being analyzed (PI: J. Rush). One goal of the trial is construct good treatment sequences for patients suffering from treatment resistant depression. www.star-d.org

41. 41