1. Outcomes Informed Care An introduction to concepts, research, and practical applications G.S. (Jeb) Brown, Ph.D. Center for Clinical Informatics Contributors : Carson Graves & Christa Castaneda; Regence

2. Top 10 reasons not to measure outcomes 10. Changes in the patients’ self report of frequency/severity of symptoms is not a valid measure of outcome. 9. Therapists will cheat and submit phony data. 8. Patients don’t like to complete questionnaires. 7. Our patients are different. 6. Too busy doing other more important things (paper work, etc).

3. Top 10 reasons not to measure outcomes 5. Insurance company will use information to deny benefits. 4. Insurance company will use information to harm therapists. 3. It is a violation of patient rights. 2. We practice evidence based medicine so we know our outcomes are good.

4. And the # 1 reason is There is no benefit to the patient or clinician!

5. What does the research show? Decades of research have shown that psychotherapy is effective for a variety of conditions. Likewise, decades of research have shown that medication is effective for a variety of conditions. Sometimes, combination treatment has seemed to be even more effective.

6. But… …There is great variability in outcomes from one patient to next, whether the treatment is Psychotherapy Medication Combination treatment. What accounts for the variance in outcomes?

7. Furthermore… Decades of research support the assertion that different methods of psychotherapy produce similar results. Likewise, different psychotropic medications may produce similar results. How to choose one? What accounts for the variance in outcomes?

8. The Dodo Bird Effect 1-9 Rosenzweig S. (1936) Some implicit common factors in diverse methods of psychotherapy: “At last the Dodo said, ‘Everybody has won and all must have prizes.’” Am J Orthopsychiatry 6:412-5.

9. Meta-analysis & common factors Three decades of meta-analytic studies have served to reinforce Rosenzweig’s 1936 observation. Lack of evidence for specific treatment effects bolster the argument that almost all of the effects of psychotherapy are due to factors common to all psychotherapies. No evidence that effect sizes have increased over the past three decades of psychotherapy/pharmacy research!!!!

10. Meta-analysis and placebo  Meta-analysis of multiple studies of antidepressants let us estimate the relative importance of common factors (placebo effects) versus drug effects. 10-12  Placebo effects are a major portion of the measured improvement.  Studies reporting large effect sizes for medication also had large effect sizes for placebos.

11. Drug effect accounts for 25% of measured improvement

12. In the last decade… Meta-analyses reveal that the clinician is more important than the technique in the variance in psychotherapy outcomes. 20-32 Some recent analyses show the prescribing clinician is at least as important as the drug in pharmacotherapy outcomes. Other recent analyses suggest the psychotherapist effects the impact of medication in combination treatment!

13. Clinician as active ingredient Traditional analysis of variance assumes the clinician does not matter. Hierarchical linear modeling specifies the clinician as a variable and possible source of variance. Reanalysis of NIMH data: clinician is primary source of variance. (Bolt, Dong-Min & Wampold, 2006) and (McKay, Imel & Wampold, 2006) Analysis of managed care data: therapist is primary source of variance. (Wampold and Brown, 2005)

14. Cross validation analysis  Psychotherapists in national managed care network ranked based on all cases from 1999- 2002 if sample size =>30  If a therapist’s mean residualized final score < 0 then clinician rated “Highly effective”; else clinician rated “Less effective”.  Outcomes evaluated in the 2003-2004 cross validation period for a new sample of cases.

15. Cross validation results

16. What the heck is effect size?  An effect size of 1 means the client improved one standard deviation on the outcome measure.  An effect size of.8 or higher is considered large.  Meta-analyses of large sample of psychotherapy studies suggest the effect size for psychotherapy is approximately.8.  There is no evidence that effect sizes have increased over past 30 years!

17. Again, why measure outcomes? A large, growing body of research over the past decade suggests that routine measurement of outcomes leads to improved outcomes, particularly for those patients most at risk. 26-35

18. What improves, what doesn’t: + Cases at risk for negative outcome -- Clinician ability to identify failing cases (M. Lambert, Personal communication) Feedback must become a routine part of clinical practice!

19. And so we conclude… In behavioral healthcare, focusing on the treatment (therapy technique, medication) is not enough to optimize outcomes – we must also focus on the outcomes for each patient and the skill of the clinician!

20. What should be validated? The Psychotherapist! “We are at the origin of a revolution of using outcomes to inform practice and guide management of services. Care must be taken to use this precious data to benefit patients rather than curtail costs. In the end, we (practitioners, researchers and third parties) should be united in our desire to optimize the benefits that psychotherapy can provide to patients. To do so, we must emphasize those aspects of psychotherapy that account for the variability in outcomes. At this point, the evidence indicates that the psychotherapists is critical.” (page 208) Bruce Wampold, PhD In Evidence-Based Practices in Mental Health; Norcross, Beutler & Levant (Eds), 2006

21. Outcomes informed clinicians… 1.Recognize the importance of clinician skill in providing effective treatments. 2.Support the desire to improve outcomes by actively evaluating them, and applying the feedback to the treatment. 3.Believe that patients benefit when referred to clinicians with practice-based evidence of effectiveness.

22.  www.psychoutcomes.org www.psychoutcomes.org  Non-profit set up to encourage the use of client/patient-completed outcome measures in behavioral health care and related fields.  TWiki site provides information, fosters collaboration, and offers support for organizations launching and nurturing outcomes-informed care initiatives.

25. The truth about outcomes questionnaires!  All patient self report outcome questionnaires tend to load on a common factor: “global distress”.  Due to the high degree of correlation between items, well constructed questionnaires of 10-15 items can have coefficients of reliability and construct validity comparable to measures of 30 or more items.  Even ultra brief questionnaires of 4-9 items may have adequate reliability and validity for must measurement needs.

26. Measurement 2.0 Measurement 1.0 0 Reliance on copyrighted and published questionnaires Copyright holder may charge fees for the use of questionnaires Copyright holder may place conditions or restrictions on the use of questionnaires Measurement 2.0  Item banks and resulting questionnaires belong to community of users.  No fees for questionnaires constructed from items in the shared item bank  Each organization is responsible for their own measurement methods and determines the appropriate content and use of questionnaires

27. Measurement 2.0 Questionnaire Development Measurement 1.0 0 A pool of items are tested in various samples Item analysis used to select items for final questionnaire Questionnaire validated, usually in correlation studies with questionnaires measuring the same construct Questionnaire published in final form  Manual published Many years may pass before a new version is published Measurement 2.0  A pool of items are tested in various samples  Item analysis used to select items for multiple versions of the questionnaires, depending on the needs of the users  Construct validity determined by factor analysis and comparison of results to known constructs. Various organizations may conduct correlation studies to satisfy internal skeptics.  Questionnaires are constantly evolving as data accumulates and measure needs change  Online manual constantly updated as data accumulates and needs of users dictate  Multiple versions available, with the community of users determining which versions offer the greatest utility

29. Outcomes Informed Process One-page self-report questionnaires Administer every session Fax to 800# when most convenient View data within 24 hours on secure personal webpage Discuss with patients; create a culture of feedback within each treatment.

30. How does it work? Forms are faxed to fax server at the Center for Clinical Informatics Data Center Data Center functions: 1.human verification of data to assure accuracy 2.capture and warehouse data in secure SQL Server database 3.apply advanced algorithms to score questionnaires and monitor patient progress 4.host Clinician’s Toolkit for viewing and graphing outcome data housed in the data warehouse. 5.provide a high level of customer service, user support & training

34. Therapeutic alliance  A large body of evidence suggests that the relationship and working alliance between the clinician and patient is an important factor in the outcome. 36-42  Routine use of a session rating/therapeutic alliance scale may permit clinicians to identify and repair problems in the working alliance.

35. Recommendations to clinicians  Practitioners are encouraged to make the creation and cultivation of the therapy relationship, characterized by the elements found to be demonstrably and probably effective, a primary aim of their treatment.  Practitioners are encouraged to adapt the therapy relationship to patient characteristics in ways shown to enhance outcomes.  Practitioners are encouraged to routinely monitor patients’ responses to the therapy relationship and ongoing treatment. Such monitoring leads to increased opportunities to repair alliance ruptures, improve the relationship, modify technical strategies, and avoid premature termination. Norcross & Lambert (page 218); In Evidence-Based Practices in Mental Health, Norcross, Beutler & Levant (Eds), 2006

37. Benchmarking outcomes Takuya Minami, PhD and colleagues have published a series of articles describing one benchmarking methodology. 43-45 Meta-analyses of controlled psychotherapy studies were used to establish effect size benchmarks for the treatment of depression. Outcomes for treatment of depression by PacifiCare Behavioral Health’s network providers found clinically equivalent to benchmark from clinical trials.

39. Can we improve outcomes?  Increasing the percentage of patients treated by highly effective clinicians (as identified through practice based evidence) is the most direct pathway open to a health plan seeking to improving outcomes across a large system of care.  Organizations may also improve outcomes by fostering outcomes informed care methods within the organization.

40. Western Psychological & Counseling: An outcomes informed organization! 2006 Baseline 2007 9% improvement 2008 22% improvement Effective range Close to national average for clinicians in private practice Highly Effective Range Comparable to outcomes from well controlled psychotherapy studies

41. Multiple perspectives on the question of what really makes a difference in treatment outcomes. Comment: In depth debate and dialog among top researchers in the field. Recommended reading

42. Rigorous review and analyses of controlled studies on psychotherapy outcome. Conclusion: much more variance resides with the clinician than with the treatments.

43. Comprehensive review of the research on the role of the therapeutic relationship and factors that make a difference in outcomes. Evidenced based relationships? Recommended reading

44. References 1.Rosenzweig S. 1936. Some implicit common factors in diverse methods of psychotherapy: “At last the Dodo said, ‘Everybody has won and all must have prizes.’” Am J Orthopsychiatry 6:412-5. 2.Shapiro DA & Shapiro D. 1982. Meta-analysis of comparative therapy outcome studies: A replication and refinement. Psychol Bull 92:581-604. 3.Robinson LA, Berman JS, Neimeyer RA. 1990. Psychotherapy for treatment of depression: A comprehensive review of controlled outcome research. Psychol Bull 108:30-49. 4.Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychol Bull 122:203-15. 5.Ahn H, Wampold BE. 2001. Where oh where are the specific ingredients? A meta-analysis of component studies in counseling and psychotherapy. J Counsel Psychol 48:251-7.

45. 6.Chambless DL, Ollendick TH. 2001. Empirically supported psychological interventions: Controversies and evidence. Annual Rev Psychol 52:685-716. 7.Luborsky, L., Rosenthal, R., Diguer, L., et al. 2002. The dodo bird verdict is alive and well--mostly. J. Psychotherapy Integration Vol 12(1) 32-57 8.Wampold BE. 2001. The great psychotherapy debate: Models, Methods, and Findings. Mahwah NJ: Lawrence Erlbaum Associates. 9. Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychol Bull 122:203-15. 10.Kirsch, I & Sapirstein, G. 1998. Listening to Prozac but hearing placebo: A meta analysis of antidepressant medication. Prevention & Treatment. 1, Article 0002a, No Pagination Specified References (continued)

46. 11.Kirsch, I. 2000. Are drug and placebo effects in depression additive? Biological Psychiatry 47, 733-73. 12.Kirsch, I, Moore, TJ, Scoboria, A, Nicholls, SS. 2002. The emperor's new drugs: An analysis of antidepressant medication data submitted to the U.S. Food and Drug Administration. Prevention & Treatment. 5(1), No Pagination Specified 13.Luborsky L, Crits-Christoph P, McLellan T, et al. 1986. Do therapists vary much in their success? Findings from four outcome studies. Am J Orthopsychiatry 56:501-12. 14.Crits-Christoph P, Baranackie K, Kurcias JS, et al. 1991. Meta-analysis of therapist effects in psychotherapy outcome studies. Psychother Res 1:81-91. 15.Crits-Christoph P, Mintz J. 1991. Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. J Consul Clin Psychol 59:20-6. References (continued)

47. 16.Wampold BE. 1997. Methodological problems in identifying efficacious psychotherapies. Psychother Res 7:21-43, 17.Elkin I. 1999. A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clin Psychol Sci Prac 6:10- 32. 18.Wampold BE, Serlin RC. 2000. The consequences of ignoring a nested factor on measures of effect size in analysis of variance designs. Psychol Methods 4:425-33. 19.Huppert JD, Bufka LF, Barlow DH, et al. 2001. Therapists, therapist variables, and cognitive-behavioral therapy outcomes in a multicenter trial for panic disorder. J Consul Clin Psychol 69:747-55. 20.Okiishi J, Lambert MJ, Nielsen SL, et al. 2003. Waiting for supershrink: An empirical analysis of therapist effects. Clin Psychol Psychother 10:361-73. References (continued)

48. 21.Brown GS, Jones ER, Lambert MJ, et al. 2005. Identifying highly effective psychotherapists in a managed care environment. Am J Managed Care 11(8):513-20. 22.Wampold BE, Brown GS. 2005. Estimating variability in outcomes due to the therapist: A naturalistic study of outcomes in managed care. J Consul Clin Psychol. 73(5): 914-923. 23.Elkin, I, Shae, T, Watkins, JT., et al. 1989. National Institute of Mental Health Treatment of Depression Collaborative Research Program: General effectiveness of treatments. Archive of General Psychiatry. 46: 971-982. 24.Kim DM, Wampold BE, Bolt DM. 2006. Therapist effects and treatment effects in psychotherapy: Analysis of the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychother Res. 16(2): 161-172. 25.McKay, KM, Imel, ZE & Wampold, BE. In press. Psychiatrist effects in the pharmacological treatment of depression. J. Affective Disorders. References (continued)

49. 26.Hannan C, Lambert MJ, Harmon C et al. 2005. A lab test and algorithms for identifying clients at risk for treatment failure. J Clin Psychol 61(2):155-63. 27.Lambert MJ, Harmon C, Slade K et al. 2005. Providing feedback to psychotherapists on their patients progress: Clinical results and practice suggestions J Clin Psychol 61(2):165-74. 28.Harmon C, Hawkins, Lambert MJ et al. 2005. Improving outcomes for poorly responding clients: The use of clinical support tools and feedback to clients. J Clin Psychol 61(2):175-85. 29.Brown GS, Jones DR. 2005. Implementation of a feedback system in a managed care environment: What are patients teaching us? J Clin Psychol 61(2):187-98. 30. Claiborn CD, Goodyear EK. 2005. Feedback in psychotherapy. J Clin Psychol 61(2):209-21. References (continued)

50. 31.Lueger RJ. 1998. Using feedback on patient progress to predict the outcome of psychotherapy. J Clin Psychol 54:383-93. 32.Lambert MJ, Whipple JL, Smart DW, et al. 2001. The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychother Res 11(1):49-68. 33.Lambert MJ, Whipple JL, Vermeersch DA, et al. 2002. Enhancing psychotherapy outcomes via providing feedback on client progress: A replication. Clin Psychol Psychother 9:91-103. 34.Whipple JL, Lambert MJ, Vermeersch DA, et al. 2003. Improving the effects of psychotherapy: The use of early identification of treatment failure and problem-solving strategies in routine practice. J Counsel Psychol 50(1):59-68. 35.Lambert MJ, Whipple JL, Hawkins EJ, et al. 2003. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clin Psychol Sci Prac 10:288-301. References (continued)

51. 36.Bachelor, A., & Horvath, A. (1999). The therapeutic relationship. In M.A. Hubble, B.L. Duncan, and S.D. Miller (eds.). The Heart and Soul of Change: What Works in Therapy. Washington, D.C.: APA Press, 133- 178. 37.Blatt, S. J., Zuroff, D.C., Quinlan, D.M., & Pilkonis, P. (1996). Interpersonal factors in brief treatment of depression: Further analyses of the NIMH Treatment of Depression Collaborative Research Program. J Consul Clin Psychol. 64, 162-171. 38.Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research, and Practice, 16, 252-260. 39.Burns, D., & Nolen-Hoeksema, S. (1992). Therapeutic empathy and recovery from depression in cognitive-behavioral therapy: A structural equation model. J Consul Clin Psychol. 60, 441-449. 40.Connors, GJ, DiClemente, CC., Carroll, KM, et al. 1997 The therapeutic alliance and its relationship to alcoholism treatment participation and outcome. J Consul Clin Psychol, 65(4), 588-598.

52. 41.Horvath, A. O., & Symonds, B. D. (1991). Relation between working alliance and outcome in psychotherapy: A meta-analysis. J Consul Clin Psychol. 38, 139-149. 42.Krupnick, J., Sotsky, SM, Simmens, S et al. (1996) The role of the therapeutic alliance in psychotherapy and pharmacotherapy outcome: Findings in the National Institute of Mental Health Treatment of Depression Collaborative Research Project. J Consul Clin Psychol., 64, 532-539. 43.Minami, T., Serlin, R. C., Wampold, B. E., Kircher, J. C., & Brown, G. S. (In press). Using clinical trials to benchmark effects produced in clinical practice, Quality and Quantity 44.Minami, T., Wampold, B. E., Serlin, R. C., Hamilton, E., Brown, G. S., & Kircher, J. (2007). Benchmarking the effectiveness of psychotherapy treatment for adult depression in a managed care environment. 45.Minami, T., Wampold, B. E., Serlin, R. C., Kircher, J. C., & Brown, G. S. (2007). Benchmarks for psychotherapy efficacy in adult major depression, Journal of Consulting and Clinical Psychology, 75, 232-243. References (continued)