1. Becoming An Outcomes Informed Clinician G.S. (Jeb) Brown, Ph.D. Center for Clinical Informatics

2. An outcomes informed clinician… Uses the best available data on treatment outcomes to inform the treatment for each client/patient Stays current on the latest research on what makes a difference in treatment outcomes. Recognizes the importance of clinician skill in providing effective treatments. Accepts personal responsibility for evaluating and improving his or her outcomes.

3. Sources of outcome data… Clinical trials designed to draw causal conclusions regarding the efficacy of various treatments. Meta analyses designed to draw conclusions based on a large sample of clinical trials. Studies evaluating the effectiveness of treatments delivered in the real world without experimental controls. Data from your own clients.

4. Clinical trials Random assignment and experimental controls designed to control of sources of variance in outcomes.  Patient factors, treatment method, dose, duration, etc. Double blind placebo controlled considered the “gold standard” Use of analysis of variance to determine if differences between treatments are “statistically significant”, i.e. unlikely to occur by chance alone.

5. Analysis of variance Statistical procedures used to analyze data from clinical trails provides an estimate of the impact of each factor to the eventual outcome of care. Analysis of variance can only calculate the variance for those factor specified in the hypotheses or “model” Warning: Failure to specify all sources of variance in the model may lead to erroneous and misleading findings.

6. Cautionary tale The NIMH funded Treatment of Depression Collaborative Research Project is one of the largest studies of the treatment of depression over conducted. 1 Evaluated outcomes for Cognitive Behavioral Therapy (CBT), Interpersonal Therapy (IT), Placebo, and Anti-depressant Therapy. 40+ articles published in peer reviewed journals Most failed to account for the single largest source of variance… the clinician

7. Initial findings Cognitive/behavioral treatment (CBT) and interpersonal therapy (IPT) were found to produce comparable benefits to the depressed patients treated 2 Imipramine found superior to Placebo 2 Tendency for IPT to be superior for the treatment of patients with more severe depression. 3

8. Critique Traditional analysis of variance evaluating treatments as the primary source of variance is correct only if the researcher is sure that the clinician does not matter! Psychotherapy research shows that the clinician matters…. A lot! 4-11 If the clinician may be a source of variance, then it is necessary to use a hierarchical linear model which specifies the clinician as a variable and possible source of variance.

9. Reanalyzes – using HLM No difference between CBT and IT for severe depression 12  0% of variance due to treatment method; 8% due to clinicians Reanalysis of placebo - imipramine comparison performed including the 9 psychiatrists as a variable 13  3.4% of variance due to medication; 9.1% due to psychiatrist  Top third of psychiatrists had a better outcome with placebo than the bottom third with imipramine

10. Clinical trials – feedback studies Michael Lambert, PhD and colleagues at the Brigham Young Comprehensive Clinic conducted a series of controlled studies investigating whether providing clinician’s feedback the client’s trajectory of change would improve outcomes. 14-19 Feedback found to significant reduce early dropout and treatment failures. Clinicians’ judgment alone, in the formed in absence of information from the questionnaires, proved to be a poor predictor treatment failures.

11. Evidenced based psychotherapy For several decades psychotherapy researchers have attempted to design randomly controlled trails (RCT) to investigate the effectiveness of specific methods of psychotherapy. Study design analogous to pharmacy trials, except that designing credible “placebo treatments” is much more problematic. Various treatment methods are being touted as “evidenced based” by citing the number of RCTs providing evidence that the treatment exceeded placebo (or some other treatment).

12. Psychotherapy “brands” The advocacy for the use of specific therapies is analogous to the advertising of brands of antidepressant medication. Calls for wide spread use of “evidence based treatments” in psychotherapy is analogous to the FDA’s insistence that a drug may not be marketed for the treatment of depression until at least two studies have shown superiority to placebo. Advocates and practitioners of various “evidence based treatments” have a vested interest in discouraging the use of “unproven” treatments.

13. Brand differentiation Advocates of psychotherapy brands insist on the uniqueness of their therapy and the need to adhere to specific treatment procedures Research methodology requires the use of manuals and other techniques to standardize treatments Treatment effectiveness presumed to be dependent on the correct application of the “active ingredients” in the psychotherapy method.

14. Recommended reading Rigorous review and analyses of controlled studies on psychotherapy outcome. Conclusion: much more variance resides with the clinician than with the treatments.

15. The Dodo Bird Effect 20-27 Rosenzweig S. (1936) Some implicit common factors in diverse methods of psychotherapy: “At last the Dodo said, ‘Everybody has won and all must have prizes.’” Am J Orthopsychiatry 6:412-5.

16. Meta-analysis & common factors Over two decades of meta-analytic studies have served to reinforce Rosenzweig’s 1936 observation that different methods of psychotherapy tend to produce comparable outcomes… the “Dodo Bird Effect” Lack of evidence for specific treatment effects bolster the argument that almost all of the effects of psychotherapy are due to factors common to all psychotherapies. 20-27

17. Meta-analyses and placebo Meta-analysis of multiple studies of antidepressants let us estimate the relative importance of common factors (placebo effects) versus drug effects. 28-30 Placebo effects are a major portion of the measured improvement. Studies reporting large effect sizes for medication also had large effect sizes for placebos. Similar to findings for psychiatrists from TDCRP data.

18. Drug effect accounted for 25% of measured improvement

19. Effectiveness studies Effectiveness studies attempt to evaluate outcomes in real world treatment settings. Heterogeneous outpatient treatment populations require use of statistical methods for case mix adjustment in order to compare results across sites or clinicians. Outcomes “benchmarking” refers to the practice of comparing outcomes from one sample to outcomes of another sample used for comparison purposes.

20. Benchmarking outcomes Takuya Minami, PhD and colleagues have published a series of articles describing one benchmarking methodology. 31-33 Meta-analyses of controlled psychotherapy studies used to establish effect size benchmarks for the treatment of depression. Outcomes for treatment of depression by PacifiCare Behavioral Health’s network providers found clinical equivalent to benchmark from clinical trials.

21. PBH Outcomes for Depression

22. Therapists and Medications Analysis of PacifiCare Behavioral Health (PBH) data reveals 5% of variance due to therapist for patients receiving psychotherapy alone. 35 % of variance due the therapist than those receiving psychotherapy in combination with a medication! 34

23. Cross validation analysis Psychotherapists in PBH network ranked based on all cases from 1999-2002 if sample size =>30; N=116. If a therapist’s mean residualized final score < 0 then clinician rated “Highly effective”; else clinician rated “Less effective”. Outcomes evaluated in the 2003-2004 cross validation period for a new sample of cases.

24. Cross validation results

25. Can we improve outcomes? Increasing the percentage of patients treated by highly effective clinicians (as identified through practice based evidence) is the most direct pathway open to a health plan seeking to improving outcomes across a large system of care. Organizations may improve outcomes by fostering outcomes informed care methods within the organization. Resources for Living and Accountable Behavioral Healthcare Alliance provide two examples using the ORS.

26. RFL results Baseline period Training and feedback

27. Accountable Behavioral HealthCare Alliance

28. Therapeutic alliance A large body of evidence suggests that the relationship and working alliance between the clinician and patient is an important factor in the outcome. 35-41 Routine use of a session rating/therapeutic alliance scale may permit clinicians to identify and repair problems in the working alliance.

29. SRS Use and Outcomes

30. SRS Alliance – positively skewed SRS=39-40 SRS=35-38 SRS=0-34

31. SRS Change and Outcomes

32. Condemnation with faint praise

33. Do outcomes informed clinicians get better outcomes? United Behavioral Health surveyed clinicians regarding their use of outcome questionnaires and outcome reports sent by the managed care company that monitored patient progress. 42 Results showed that clinicians who reported using outcome information had patients who also reported greater improvement at 6 months from baseline. Improvement per session was greatest among patients whose clinicians reported reading the outcome report and using outcome measures in their clinical practice.

34. Regence Outcomes for Depression Regence solicited self identified “outcomes informed clinicians” to participate in an outcomes informed care pilot project.

35. Implications for clinicians Good news: The clinician matters!!!!!! All treatments (including medications!?) are only as effective as the clinicians delivering the treatment. Psychotherapy is profession requiring a high degree of skill. Expert skill is acquired through practice and performance feedback. Routine use of outcome and alliance questionnaires provide clinicians the means to evaluate and improve their skill and effectiveness.

36. References 1.Elkin, I., Parloff, M.B., Hadley, S.W., Autry, J.H., 1985. NIMH treatment of depression collaborative research program: background and research plan. Archives of General Psychiatry 42, 305–316. 2.Elkin, I., Shea, T., Watkins, J.T., Imber, S.D., Sotsky, S.M., Collins, J.F., et al., 1989. National institute of mental health treatment of depression collaborative research program: general effectiveness of treatments. Archives of General Psychiatry 46, 971–982. 3.Elkin, I., Gibbons, R.D., Shea, M.T., Sotsky, S.M., Watkins, J.T., Pilkonis, P.A., et al., 1995. Initial severity and differential treatment outcome in the National Institute of Mental Health treatment of depression collaborative research program. Journal of Consulting and Clinical Psychology 63, 841–847. 260 4.Luborsky L, Crits-Christoph P, McLellan T, et al. 1986. Do therapists vary much in their success? Findings from four outcome studies. Am J Orthopsychiatry 56:501-12.

37. References (continued) 5.Crits-Christoph P, Baranackie K, Kurcias JS, et al. 1991. Meta-analysis of therapist effects in psychotherapy outcome studies. Psychother Res 1:81-91. 6.Crits-Christoph P, Mintz J. 1991. Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. J Consul Clin Psychol 59:20-6. 7.Wampold BE. 1997. Methodological problems in identifying efficacious psychotherapies. Psychother Res 7:21-43, 8.Elkin I. 1999. A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clin Psychol Sci Prac 6:10- 32. 9.Wampold BE, Serlin RC. 2000. The consequences of ignoring a nested factor on measures of effect size in analysis of variance designs. Psychol Methods 4:425-33. 10.Huppert JD, Bufka LF, Barlow DH, et al. 2001. Therapists, therapist variables, and cognitive-behavioral therapy outcomes in a multicenter trial for panic disorder. J Consul Clin Psychol 69:747-55.

38. References (continued) 11.Okiishi J, Lambert MJ, Nielsen SL, et al. 2003. Waiting for supershrink: An empirical analysis of therapist effects. Clin Psychol Psychother 10:361-73. 12.Kim DM, Wampold BE, Bolt DM. 2006. Therapist effects and treatment effects in psychotherapy: Analysis of the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychother Res. 16(2): 161-172. 13.McKay, K. M., Imel, Z. E., & Wampold, B. E. (2006). Psychiatrist effects in the psychopharmacological treatment of depression. Journal of Affective Disorders, 92, 287-290. 14.Lueger RJ. 1998. Using feedback on patient progress to predict the outcome of psychotherapy. J Clin Psychol 54:383-93. 15.Lambert MJ, Whipple JL, Smart DW, et al. 2001. The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychother Res 11(1):49-68.

39. References (continued) 16.Hannan C, Lambert MJ, Harmon C et al. 2005. A lab test and algorithms for identifying clients at risk for treatment failure. J Clin Psychol 61(2):155-63. 17.Lambert MJ, Harmon C, Slade K et al. 2005. Providing feedback to psychotherapists on their patients progress: Clinical results and practice suggestions J Clin Psychol 61(2):165-74. 18.Harmon C, Hawkins, Lambert MJ et al. 2005. Improving outcomes for poorly responding clients: The use of clinical support tools and feedback to clients. J Clin Psychol 61(2):175-85. 19.Brown GS, Jones DR. 2005. Implementation of a feedback system in a managed care environment: What are patients teaching us? J Clin Psychol 61(2):187-98. Claiborn CD, Goodyear EK. 2005. Feedback in psychotherapy. J Clin Psychol 61(2):209-21. 20.Rosenzweig S. 1936. Some implicit common factors in diverse methods of psychotherapy: “At last the Dodo said, ‘Everybody has won and all must have prizes.’” Am J Orthopsychiatry 6:412-5.

40. References (continued) 21.Shapiro DA & Shapiro D. 1982. Meta-analysis of comparative therapy outcome studies: A replication and refinement. Psychol Bull 92:581-604. 22.Robinson LA, Berman JS, Neimeyer RA. 1990. Psychotherapy for treatment of depression: A comprehensive review of controlled outcome research. Psychol Bull 108:30-49. 23.Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychol Bull 122:203-15. 24.Ahn H, Wampold BE. 2001. Where oh where are the specific ingredients? A meta-analysis of component studies in counseling and psychotherapy. J Counsel Psychol 48:251-7. 25.Chambless DL, Ollendick TH. 2001. Empirically supported psychological interventions: Controversies and evidence. Annual Rev Psychol 52:685- 716. 26. Luborsky, L., Rosenthal, R., Diguer, L., et al. 2002. The dodo bird verdict is alive and well--mostly. J. Psychotherapy Integration Vol 12(1) 32-57

41. References (continued) 27.Wampold BE. 2001. The great psychotherapy debate: Models, Methods, and Findings. Mahwah NJ: Lawrence Erlbaum Associates. Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychol Bull 122:203-15. 28.Kirsch, I & Sapirstein, G. 1998. Listening to Prozac but hearing placebo: A meta analysis of antidepressant medication. Prevention & Treatment. 1, Article 0002a, No Pagination Specified 29.Kirsch, I. 2000. Are drug and placebo effects in depression additive? Biological Psychiatry 47, 733-73. 30.Kirsch, I, Moore, TJ, Scoboria, A, Nicholls, SS. 2002. The emperor's new drugs: An analysis of antidepressant medication data submitted to the U.S. Food and Drug Administration. Prevention & Treatment. 5(1), No Pagination Specified

42. References (continued) 31.Minami, T., Serlin, R. C., Wampold, B. E., Kircher, J. C., & Brown, G. S. (In press). Using clinical trials to benchmark effects produced in clinical practice, Quality and Quantity 32.Minami, T., Wampold, B. E., Serlin, R. C., Hamilton, E., Brown, G. S., & Kircher, J. (2007). Benchmarking the effectiveness of psychotherapy treatment for adult depression in a managed care environment. 33.Minami, T., Wampold, B. E., Serlin, R. C., Kircher, J. C., & Brown, G. S. (2007). Benchmarks for psychotherapy efficacy in adult major depression, Journal of Consulting and Clinical Psychology, 75, 232-243. 34.Wampold BE, Brown GS. 2005. Estimating variability in outcomes due to the therapist: A naturalistic study of outcomes in managed care. J Consul Clin Psychol. 73(5): 914-923. 35.Bachelor, A., & Horvath, A. (1999). The therapeutic relationship. In M.A. Hubble, B.L. Duncan, and S.D. Miller (eds.). The Heart and Soul of Change: What Works in Therapy. Washington, D.C.: APA Press, 133-178.

43. References (continued) 36.Blatt, S. J., Zuroff, D.C., Quinlan, D.M., & Pilkonis, P. (1996). Interpersonal factors in brief treatment of depression: Further analyses of the NIMH Treatment of Depression Collaborative Research Program. J Consul Clin Psychol. 64, 162-171. 37.Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research, and Practice, 16, 252-260. 38.Burns, D., & Nolen-Hoeksema, S. (1992). Therapeutic empathy and recovery from depression in cognitive-behavioral therapy: A structural equation model. J Consul Clin Psychol. 60, 441-449. 39.Connors, GJ, DiClemente, CC., Carroll, KM, et al. 1997 The therapeutic alliance and its relationship to alcoholism treatment participation and outcome. J Consul Clin Psychol, 65(4), 588-598. 40.Horvath, A. O., & Symonds, B. D. (1991). Relation between working alliance and outcome in psychotherapy: A meta-analysis. J Consul Clin Psychol. 38, 139-149.

44. References (continued) 41.Krupnick, J., Sotsky, SM, Simmens, S et al. (1996) The role of the therapeutic alliance in psychotherapy and pharmacotherapy outcome: Findings in the National Institute of Mental Health Treatment of Depression Collaborative Research Project. J Consul Clin Psychol., 64, 532-539. 42.Azocar, F., Cuffel, B, McCulloch, J., McCabe, J., Tani, S., Brodey, B. (2007) Monitoring patient improvement and treatment outcomes in managed behavioral healthcare. Journal for Healthcare Quality (March/April)

45. About the presenter G.S. (Jeb) Brown is a licensed psychologist with a Ph.D. from Duke University. He served as the Executive Director of the Center for Family Development from 1982 to 19987. He then joined United Behavioral Systems (an United Health Care subsidiary) as the Executive Director for of Utah, a position he held for almost six years. In 1993 he accepted a position as the Corporate Clinical Director for Human Affairs International (HAI), at that time one of the largest managed behavioral healthcare companies in the country. In 1998 he left HAI to found the Center for Clinical Informatics, a consulting firm specializing in helping large organizations implement outcomes management systems. Client organizations include Resources for Living, Regence, United Behavioral Health, Accountable Behavioral Health Care Alliance, and assorted treatment centers. Dr. Brown continues to work as a part time psychotherapist at behavioral health clinic in Salt Lake City, Utah. He does measure his outcomes.