1. Implications of Therapist Effects for Employers and Health Plans American Psychological Association Convention San Francisco, August, 2007. G.S. (Jeb) Brown, Ph.D. Center for Clinical Informatics

2. Therapists effects Wampold and others argue that researchers have ignored the individual therapist as a source of variance. 11, 16-24 The person of the therapist is necessary to delivery the treatment, and personal characteristics of the therapist modify the effect of the treatment. Factors contributing to therapists effects may include elements clinical skill and knowledge as well as personality traits.

3. Variance due to the clinician Published research making use of HLM points to the conclusion that the clinician accounts for much more of the variance in psychotherapy outcomes that treatment method per se. 11, 17-21 Analyses of PacifiCare Behavioral Health’s massive database database on patient outcomes confirms significant variance in psychotherapy outcomes at the clinician level. 24,25

4. PBH research collaboration PBH actively sought the involvement of leading psychotherapy outcomes researchers from leading academic institutions. External researchers actively involved in design of the measurement system and ongoing analysis of the data. PBH encouraged publication of findings in academic journals.

5. The (almost) Bell Curve PBH data Solo clinicians with sample sizes => 20

6. Therapists and meds Outcomes (residualized scores) of 15 therapists for patients with concurrent medication or no medication 25

7. Cross validation analysis Psychotherapists in PBH network ranked based on all cases from 1999-2002 if sample size =>30; N=116. If a therapist’s mean residualized final score < 0 then clinician rated “Highly effective”; else clinician rated “Less effective”. Outcomes evaluated in the 2003-2004 cross validation period for a new sample of cases.

8. Cross validation results

9. Psychiatrist effects Wampold and colleagues also used HLM to reanalyze the results antidepressant and placebo legs of the TDCRP study. 28 Included the 9 individual psychiatrists as a variable. Outcome measured by change on patient self report measure (Beck Depression Inventory). 9.1% of the variance due to the psychiatrist; only 3.4% due to the medication. Top third of psychiatrists achieved a better outcome with placebo than bottom third achieved with the antidepressant.

10. Honors for Outcomes Selection Criteria: –Minimum of 10 cases with two Y/LSQ data points in past 3 years –Average patient change must be reliably above average: 65% confidence that the provider’s Change Index >0 –Change Index is a case-mix adjusted measure, compares outcomes to PBH’s large normative database Honors for Outcomes is updated quarterly Honor for Outcomes

11. Website

12. Outcomes and cost

13. Value Index Value Index = Average effect size per $1000 expenditure (Effect Size/Cost of Care) x $1000

14. Implications for clinicians Good news: The clinician matters!!!!!! All treatments (including medications!?) are only as effective as the clinicians delivering the treatment. Clinicians have an ethical responsibility to assess and improve their personal effectiveness as clinicians… they cannot rely on the treatments alone to be curative. Effective clinicians deliver high value services and are worth more money!!!

15. Implications for employers and health plans Strong business case can be made for identifying “high value outcomes informed clinicians” Steering referrals to highly effective clinicians results in better treatment outcomes without increasing cost. Use of highly effective clinicians leads to greater “return on investment” for spending on behavioral healthcare services  Workplace productivity gains  Reduce medical costs

16. Barriers to identifying effective clinicians Clinician resistance to use of outcome questionnaires  “The questionnaires don’t give me any information I don’t already know.”  “I don’t believe the questionnaires provides a valid measure of outcome.” Over reliance on “evidence based treatments” and medications  “I know I get good outcomes because I am an expert in how to provide __________ treatment.” Active resistance from professional guild organizations Inability of a health plan to collect data on more than a small percentage of a clinician’s case load

17. Winners and losers? Four major categories of “stake holders”  Consumers/Patients  Highly effective clinicians  Ineffective clinicians  Payers: Employers and health plans Who benefits if outcomes informed care become widespread? Who benefits if professional guilds and practitioners continue to resist outcomes informed care?

18. Something different…. Major employer in the Pacific Northwest has partnered with Regence to implement an outcomes informed care pilot project Regence project seeks to identify and promote “outcomes informed clinicians” Clinicians are encouraged to use questionnaires with all patients and are given access to all of their outcome data in near real time. Clinician participation is entirely voluntary and at no cost to the clinician.

19. A C ollaborative O utcomes R esource N etwork ACORN Organization Inc. is a non-profit organization devoted to furthering the science and practice of outcomes informed care. ACORN maintains a TWiki which which provides information on freely available questionnaires and methods for benchmarking outcomes. ACORN site utilized by a number of organizations to support outcomes informed care initiatives. www.psychoutcomes.org

20. References & suggested readings 1.Kirsch, I & Sapirstein, G. 1998. Listening to Prozac but hearing placebo: A meta analysis of antidepressant medication. Prevention & Treatment. 1, Article 0002a, No Pagination Specified 2.Kirsch, I. 2000. Are drug and placebo effects in depression additive? Biological Psychiatry 47, 733-73. 3.Kirsch, I, Moore, TJ, Scoboria, A, Nicholls, SS. 2002. The emperor's new drugs: An analysis of antidepressant medication data submitted to the U.S. Food and Drug Administration. Prevention & Treatment. 5(1), No Pagination Specified 4. Rosenzweig S. 1936. Some implicit common factors in diverse methods of psychotherapy: “At last the Dodo said, ‘Everybody has won and all must have prizes.’” Am J Orthopsychiatry 6:412-5. 5.Shapiro DA & Shapiro D. 1982. Meta-analysis of comparative therapy outcome studies: A replication and refinement. Psychol Bull 92:581-604.

21. References & suggested readings 6.Robinson LA, Berman JS, Neimeyer RA. 1990. Psychotherapy for treatment of depression: A comprehensive review of controlled outcome research. Psychol Bull 108:30-49. 7.Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychol Bull 122:203-15. 8.Ahn H, Wampold BE. 2001. Where oh where are the specific ingredients? A meta-analysis of component studies in counseling and psychotherapy. J Counsel Psychol 48:251-7. 9.Chambless DL, Ollendick TH. 2001. Empirically supported psychological interventions: Controversies and evidence. Annual Rev Psychol 52:685-716. 10.Luborsky, L., Rosenthal, R., Diguer, L., et al. 2002. The dodo bird verdict is alive and well--mostly. J. Psychotherapy Integration Vol 12(1) 32-57

22. References & suggested readings 11.Wampold BE. 2001. The great psychotherapy debate: Models, Methods, and Findings. Mahwah NJ: Lawrence Erlbaum Associates. Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychol Bull 122:203-15. 12.McCall, WA. 1923 How to experiment in education. New York: McmIllan. 13.Fisher, RA. 1935 The design of experiments. Edinburgh: Oliver and Boyd. 14.Gehan, E. & Lemark, NA. 1994. Statistics in medical research: Developments in clinical trials. New York: Plenum Press. 15.Martindale C. 1978. The therapist-as-fixed-effect fallacy in psychotherapy research. J Consult Clin Psychol 46:1526-30.

23. References (continued) 16.Luborsky L, Crits-Christoph P, McLellan T, et al. 1986. Do therapists vary much in their success? Findings from four outcome studies. Am J Orthopsychiatry 56:501-12. 17.Crits-Christoph P, Baranackie K, Kurcias JS, et al. 1991. Meta- analysis of therapist effects in psychotherapy outcome studies. Psychother Res 1:81-91. 18.Crits-Christoph P, Mintz J. 1991. Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. J Consul Clin Psychol 59:20-6. 19.Wampold BE. 1997. Methodological problems in identifying efficacious psychotherapies. Psychother Res 7:21-43, 20.Elkin I. 1999. A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clin Psychol Sci Prac 6:10- 32.

24. References (continued) 21.Wampold BE, Serlin RC. 2000. The consequences of ignoring a nested factor on measures of effect size in analysis of variance designs. Psychol Methods 4:425-33. 22.Huppert JD, Bufka LF, Barlow DH, et al. 2001. Therapists, therapist variables, and cognitive-behavioral therapy outcomes in a multicenter trial for panic disorder. J Consul Clin Psychol 69:747-55. 23.Okiishi J, Lambert MJ, Nielsen SL, et al. 2003. Waiting for supershrink: An empirical analysis of therapist effects. Clin Psychol Psychother 10:361-73. 24.Brown GS, Jones ER, Lambert MJ, et al. 2005. Identifying highly effective psychotherapists in a managed care environment. Am J Managed Care 11(8):513-20. 25.Wampold BE, Brown GS. 2005. Estimating variability in outcomes due to the therapist: A naturalistic study of outcomes in managed care. J Consul Clin Psychol. 73(5): 914-923.

25. References (continued) 26.Elkin, I, Shae, T, Watkins, JT., et al. 1989. National Institute of Mental Health Treatment of Depression Collaborative Research Program: General effectiveness of treatments. Archive of General Psychiatry. 46: 971-982. 27.Kim DM, Wampold BE, Bolt DM. 2006. Therapist effects and treatment effects in psychotherapy: Analysis of the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychother Res. 16(2): 161-172. 28.McKay, KM, Imel, ZE & Wampold, BE. 2006. Psychiatrist effects in the pharmacological treatment of depression. J. Affective Disorders. 92(2-3): 287-290. 29.Hannan C, Lambert MJ, Harmon C et al. 2005. A lab test and algorithms for identifying clients at risk for treatment failure. J Clin Psychol 61(2):155-63. 30.Lambert MJ, Harmon C, Slade K et al. 2005. Providing feedback to psychotherapists on their patients progress: Clinical results and practice suggestions J Clin Psychol 61(2):165-74.

26. References (continued) 31.Harmon C, Hawkins, Lambert MJ et al. 2005. Improving outcomes for poorly responding clients: The use of clinical support tools and feedback to clients. J Clin Psychol 61(2):175-85. 32.Brown GS, Jones DR. 2005. Implementation of a feedback system in a managed care environment: What are patients teaching us? J Clin Psychol 61(2):187-98. 33. Claiborn CD, Goodyear EK. 2005. Feedback in psychotherapy. J Clin Psychol 61(2):209-21. 34.Lueger RJ. 1998. Using feedback on patient progress to predict the outcome of psychotherapy. J Clin Psychol 54:383-93. 35.Lambert MJ, Whipple JL, Smart DW, et al. 2001. The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychother Res 11(1):49-68.

27. References (continued) 36.Lambert MJ, Whipple JL, Vermeersch DA, et al. 2002. Enhancing psychotherapy outcomes via providing feedback on client progress: A replication. Clin Psychol Psychother 9:91-103. 37.Whipple JL, Lambert MJ, Vermeersch DA, et al. 2003. Improving the effects of psychotherapy: The use of early identification of treatment failure and problem-solving strategies in routine practice. J Counsel Psychol 50(1):59-68. 38.Lambert MJ, Whipple JL, Hawkins EJ, et al. 2003. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clin Psychol Sci Prac 10:288-301. 39.Bachelor, A., & Horvath, A. (1999). The therapeutic relationship. In M.A. Hubble, B.L. Duncan, and S.D. Miller (eds.). The Heart and Soul of Change: What Works in Therapy. Washington, D.C.: APA Press, 133-178. 40.Blatt, S. J., Zuroff, D.C., Quinlan, D.M., & Pilkonis, P. (1996). Interpersonal factors in brief treatment of depression: Further analyses of the NIMH Treatment of Depression Collaborative Research Program. J Consul Clin Psychol. 64, 162-171.

28. References (continued) 41.Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research, and Practice, 16, 252-260. 42.Burns, D., & Nolen-Hoeksema, S. (1992). Therapeutic empathy and recovery from depression in cognitive-behavioral therapy: A structural equation model. J Consul Clin Psychol. 60, 441-449. 43.Connors, GJ, DiClemente, CC., Carroll, KM, et al. 1997 The therapeutic alliance and its relationship to alcoholism treatment participation and outcome. J Consul Clin Psychol, 65(4), 588-598. 44.Horvath, A. O., & Symonds, B. D. (1991). Relation between working alliance and outcome in psychotherapy: A meta-analysis. J Consul Clin Psychol. 38, 139-149. 45.Krupnick, J., Sotsky, SM, Simmens, S et al. 1996. The role of the therapeutic alliance in psychotherapy and pharmacotherapy outcome: Findings in the National Institute of Mental Health Treatment of Depression Collaborative Research Project. J Consul Clin Psychol., 64, 532-539.

29. About the presenter G.S. (Jeb) Brown is a licensed psychologist with a Ph.D. from Duke University. He served as the Executive Director of the Center for Family Development from 1982 to 19987. He then joined United Behavioral Systems (an United Health Care subsidiary) as the Executive Director for of Utah, a position he held for almost six years. In 1993 he accepted a position as the Corporate Clinical Director for Human Affairs International (HAI), at that time one of the largest managed behavioral healthcare companies in the country. In 1998 he left HAI to found the Center for Clinical Informatics, a consulting firm specializing in helping large organizations implement outcomes management systems. Current client organizations include Accountable Behavioral Health Care Alliance, Kaiser Permanente, Regence, Resources for Living, and United Behavioral Health. Dr. Brown also works part time as a psychotherapist and he does measure his outcomes.

30. http://www.clinical-informatics.com jebbrown@clinical-informatics.com 1821 Meadowmoor Rd. Salt Lake City, UT 84117 Voice 801-541-9720