The importance of therapist effects in the treatment of children and adolescents G.S. (Jeb) Brown, Ph.D. Center for Clinical Informatics

Overview Findings based on analyses of data collected through the PacifiCare Behavioral Health/United Behavioral Health ALERT clinical information system Novel methodology demonstrates that much of the variance in outcomes resides with the individual psychotherapist.

Therapist effects Therapist differ in their “effectiveness” resulting in wide differences in outcomes.  Results cannot be explained by theoretical orientation, treatment methods, years of training or experience. The effectiveness of all treatments, including medications, are mediated by therapist effects. Failure to measure and account for therapist effects is simply BAD SCIENCE!

Outcome Measures 30-item patient self-report questionnaires  Life Status Questionnaire (adults)  Youth Life Status Questionnaire (children) Administered at 1 st, 3 rd, 5 th sessions and every 5 th session there after Over 20,000 questionnaires per month come into the system

Case mix Case mix variables are those variables present at the beginning of the treatment episode that are predictive of the outcome Intake score accounts for 18% of variance in change scores in PBH data Addition of age, sex and diagnosis to predictive model accounts for < 2% additional variance

Where’s the variance? % of variance in change scores

Where else is the variance?

1046 Provider Locations on Honor Roll 56 Group Practices 740 Individuals (not including those in groups) These providers accounted for about 30% of PBH outpatient episodes in 2004 Honors for Outcomes

Honors for Outcomes Selection Criteria: –Minimum of 10 cases with two Y/LSQ data points in past 3 years –Average patient change must be reliably above average: 65% confidence that the provider’s Change Index >0 –Change Index is a case-mix adjusted measure, compares outcomes to PBH’s large normative database Honors for Outcomes is updated quarterly Honor Rating Method

Website

Honors for Outcomes - Search

Honors for Outcomes - Results

Honors for Outcomes depends on predictive validity of Honors rating; prior performance predicts future performance Question: Does a therapist’s outcomes with adults predict outcomes with children and adolescents? Implication if yes: Therapists’ effectiveness is likely to be global in nature rather than specific to age and or diagnostic group. Study Question 1

Question: Does a therapist’s outcomes with adults predict outcomes with children and adolescents on medications? Implication if yes: The therapist effectiveness of the therapists is apparently mediating the effect of the medication(s). Study Question 2

Use Honors for Outcomes methodology to rank clinicians based on their outcomes with adult patients only. Therapist included in the study if they treated at least one child/adolescent with psychotherapy only and one with psychotherapy plus medication. (929 Honors, 1352 Non-Honors) Compare outcomes for children and adolescents for Honors clinicians to other clinicians. Study Method

Result: Outcomes for adults predicts outcomes for children

Results after adjusting for intake score, age, sex, diagnosis and prior treatment history.

All diagnoses and medications

Children diagnosed with depression and treated with psychotherapy alone or in combination with an antidepressant

Depression & antidepressants

Implications for clinicians The effectiveness of the individual clinician is the major source of variance in outcomes. All treatments (including medications) are only as effective and the clinicians delivering the treatment. Clinicians have an ethical responsibility to assess and improve their personal effectiveness as clinicians… they cannot rely on the treatments alone to be curative.

Implications for administrators & policy makers Exclusive focus on the effectiveness of treatments rather than clinicians limits the potential to improve outcomes. Administrators and policy makers have an obligation to consumers to assure that they have access to effective clinicians. Failure to monitor outcomes at the clinician level places consumers at risk.

Meadowmoor Rd. Salt Lake City, UT Voice

Suggested readings Ahn H, Wampold BE. Where oh where are the specific ingredients? A meta-analysis of component studies in counseling and psychotherapy. Journal of Counseling Psychology; 2001: 48, Blatt SJ, Sanislow CA, Zuroff DC, Pilkonis PA. Characteristics of effective therapists: Further analyses of data from the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Journal of Consulting and Clinical Psychology; 1996: 64, Brown GS, Burlingame GM, Lambert MJ, et al. Pushing the quality envelope: A new outcomes management system. Psychiatric Services; 2001: 52 (7), Brown GS, Herman R, Jones ER, Wu J. Improving substance abuse assessments in a managed care environment. Joint Commission Journal on Quality and Safety; 2004: 30(8), Brown GS, Jones ER, Betts W, Wu J. Improving suicide risk assessment in a managed-mare environment. Crisis; 2003: 24(2), Brown GS, Jones ER, Lambert MJ, Minami T. Identifying highly effective psychotherapists in a managed care environment. American Journal of Managed Care, 2005: 11(8): Brown GS, Jones ER. Implementation of a feedback system in a managed care environment: What are patients teaching us? Clinical Psychology/In Session: 2005: 61(2), Burlingame GM, Jasper BW, Peterson G, et al. Administration and Scoring Manual for the YLSQ. Wilmington, DL, American Professional Credentialing Services; Crits-Christoph P, Mintz J. Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. Journal of Consulting and Clinical Psychology, 1991; 59, Crits-Christoph P., Baranackie K., Kurcias JS et al. Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research; 1991: 1, Elkin I. A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clinical Psychology: Science and Practice; 1999: 6,

1046 Provider Locations on Honor Roll 56 Group Practices 740 Individuals (not including those in groups) These providers accounted for about 30% of PBH outpatient episodes in 2004 Honors for Outcomes

Honors for Outcomes Selection Criteria: –Minimum of 10 cases with two Y/LSQ data points in past 3 years –Average patient change must be reliably above average: 65% confidence that the provider’s Change Index >0 –Change Index is a case-mix adjusted measure, compares outcomes to PBH’s large normative database Honors for Outcomes is updated quarterly Honor Rating Method

Website

Honors for Outcomes - Search

Honors for Outcomes - Results

Honors for Outcomes depends on predictive validity of Honors rating; prior performance predicts future performance Question: Does a therapist’s outcomes with adults predict outcomes with children and adolescents? Implication if yes: Therapists’ effectiveness is likely to be global in nature rather than specific to age and or diagnostic group. Study Question 1

Question: Does a therapist’s outcomes with adults predict outcomes with children and adolescents on medications? Implication if yes: The therapist effectiveness of the therapists is apparently mediating the effect of the medication(s). Study Question 2

Use Honors for Outcomes methodology to rank clinicians based on their outcomes with adult patients only. Therapist included in the study if they treated at least one child/adolescent with psychotherapy only and one with psychotherapy plus medication. (929 Honors, 1352 Non-Honors) Compare outcomes for children and adolescents for Honors clinicians to other clinicians. Study Method

Result: Outcomes for adults predicts outcomes for children

Results after adjusting for intake score, age, sex, diagnosis and prior treatment history.

All diagnoses and medications

Children diagnosed with depression and treated with psychotherapy alone or in combination with an antidepressant

Depression & antidepressants

Implications for clinicians The effectiveness of the individual clinician is the major source of variance in outcomes. All treatments (including medications) are only as effective and the clinicians delivering the treatment. Clinicians have an ethical responsibility to assess and improve their personal effectiveness as clinicians… they cannot rely on the treatments alone to be curative.

Implications for administrators & policy makers Exclusive focus on the effectiveness of treatments rather than clinicians limits the potential to improve outcomes. Administrators and policy makers have an obligation to consumers to assure that they have access to effective clinicians. Failure to monitor outcomes at the clinician level places consumers at risk.

Meadowmoor Rd. Salt Lake City, UT Voice

Suggested readings Ahn H, Wampold BE. Where oh where are the specific ingredients? A meta-analysis of component studies in counseling and psychotherapy. Journal of Counseling Psychology; 2001: 48, Blatt SJ, Sanislow CA, Zuroff DC, Pilkonis PA. Characteristics of effective therapists: Further analyses of data from the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Journal of Consulting and Clinical Psychology; 1996: 64, Brown GS, Burlingame GM, Lambert MJ, et al. Pushing the quality envelope: A new outcomes management system. Psychiatric Services; 2001: 52 (7), Brown GS, Herman R, Jones ER, Wu J. Improving substance abuse assessments in a managed care environment. Joint Commission Journal on Quality and Safety; 2004: 30(8), Brown GS, Jones ER, Betts W, Wu J. Improving suicide risk assessment in a managed-mare environment. Crisis; 2003: 24(2), Brown GS, Jones ER, Lambert MJ, Minami T. Identifying highly effective psychotherapists in a managed care environment. American Journal of Managed Care, 2005: 11(8): Brown GS, Jones ER. Implementation of a feedback system in a managed care environment: What are patients teaching us? Clinical Psychology/In Session: 2005: 61(2), Burlingame GM, Jasper BW, Peterson G, et al. Administration and Scoring Manual for the YLSQ. Wilmington, DL, American Professional Credentialing Services; Crits-Christoph P, Mintz J. Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. Journal of Consulting and Clinical Psychology, 1991; 59, Crits-Christoph P., Baranackie K., Kurcias JS et al. Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research; 1991: 1, Elkin I. A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clinical Psychology: Science and Practice; 1999: 6,

Suggested readings (continued) Hannan C, Lambert MJ, Harmon C, Nielsen SL, Smart DW & Shimokawa K, Sutton SW. A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology/In Session: 2005: 61(2), Harmon C, Hawkins, EJ, Lambert, MJ, Slade K & Whipple JL. Improving outcomes for poorly responding clients: the use of clinical support tools and feedback to clients. 61(2), Huppert JD, Bufka LF, Barlow DH, Gorman JM, Shear MK, Woods SW. Therapists, therapist variables, and cognitive-behavioral therapy outcomes in a multicenter trial for panic disorder. Journal of Consulting and Clinical Psychology; 2001: 69, Kim DM, Wampold BE, Bolt DM. Therapist effects and treatment effects in psychotherapy: Analysis of the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychotherapy Research: 2006: 12(2), Lambert MJ., Whipple J., Smart, DW et al (Vermeersch, D. A., Nielsen, S.L., & Hawkins, E. J.) The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychotherapy Research; 2001: 11, Lambert MJ, Harmon C, Slade K, Whipple JL & Hawkins EL. Providing feedback to psychotherapists on their patients’ progress: clinical results and practice suggestions. Journal of Clinical Psychology/In Session: 2005: 61(2), Lambert MJ, Hatfield DR, Vermeersch DA., et al. Administration and scoring manual for the LSQ (Life Status Questionnaire). East Setauket, NY: American Professional Credentialing Services; Lambert MJ, Whipple JL, Hawkins EJ et al. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clinical Psychology: Science & Practice; 2003: 10, Lambert MJ. Emerging methods for providing clinicians with timely feedback on treatment effectiveness. Journal of Clinical Psychology/In Session: 2005: 61(2),

Suggested readings (continued) Luborsky L, Crits-Christoph P, McLellan T et al. Do therapists vary much in their success? Findings from four outcome studies. American Journal of Orthopsychiatry; 1986: 56, Luborsky L, Rosenthal R, Diguer L et al. The dodo bird verdict is alive and well--mostly. Clinical Psychology: Science & Practice; 2002: 9(1) Matsumoto K, Jones E, Brown, GS. Using clinical informatics to improve outcomes: A new approach to managing behavioral healthcare. Journal of Information Technology in Health Care; 2003: 1(2), Okiishi J, Lambert MJ, Nielsen SL, Ogles BM. Waiting for supershrink: An empirical analysis of therapist effects. Clinical Psychology and Psychotherapy; 2003: 10, Porter ME & Teisberg, EO. Redefining competition in health care. Harvard Business Review,2004: Shapiro DA, Shapiro, D. Meta-analysis of comparative therapy outcome studies: A replication and refinement. Journal of consulting and Clinical Psychology; 1982: 92, 581–604. Vermeersch DA, Lambert MJ, Burlingame GM. Outcome Questionnaire: Item sensitivity to change. Journal of Personality Assessment; 2002: 74, Wampold BE, Brown GS. Estimating therapist variability: A naturalistic study of outcomes in private practice. Journal of Consulting and Clinical Psychology; 2005: 75(5) pp Wampold BE, Mondin GW, Moody M et al. A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “all must have prizes.” Psychological Bulletin; 1997: 122,

About the presenter G.S. (Jeb) Brown is a licensed psychologist with a Ph.D. from Duke University. He served as the Executive Director of the Center for Family Development from 1982 to He then joined United Behavioral Systems (an United Health Care subsidiary) as the Executive Director for of Utah, a position he held for almost six years. In 1993 he accepted a position as the Corporate Clinical Director for Human Affairs International (HAI), at that time one of the largest managed behavioral healthcare companies in the country. In 1998 he left HAI to found the Center for Clinical Informatics, a consulting firm specializing in helping large organizations implement outcomes management systems. Client organizations include PacifiCare Behavioral Health/ United Behavioral Health, Department of Mental Health for the District of Columbia, Accountable Behavioral Health Care Alliance, Resources for Living and assorted treatment programs and centers throughout the world. Dr. Brown continues to work as a part time psychotherapist at behavioral health clinic in Salt Lake City, Utah. He does measure his outcomes.