1. Introduction to Outcomes Informed Care

2. The content for this course is offered by A Collaborative Outcomes Resource Network (ACORN). https://www.psychoutcomes.org ACORN is a non-profit organization promoting Outcomes Informed Care in behavioral health care and related fields. TWiki site provides information, fosters collaboration, and offers support for organizations launching and nurturing outcomes-informed care initiatives. The organization is supported by health plans, managed care companies, and employers seeking to improve treatment outcomes in mental health care.

3. Course Content 1.Required and Recommended Readings 2.What is “Outcome Informed Care?” 3.Background Research 4.Measurement Concepts 5.Clinician’s Use of Feedback 6.Financial Implications

4. 1.Required Readings a.Outcomes Informed Care: An overviewOutcomes Informed Care: An overview b.Outcomes Measurement 2.0: Emerging technologies for managing treatment outcomes in behavioral healthcareOutcomes Measurement 2.0: Emerging technologies for managing treatment outcomes in behavioral healthcare c.ACORN Criteria for Effectiveness (ACE) Treatment Outcomes Evaluation MethodACORN Criteria for Effectiveness (ACE) Treatment Outcomes Evaluation Method d.Outcomes Measurement: Concepts and definitionsOutcomes Measurement: Concepts and definitions

5. 1.Recommended Reading This is an up-to-date review of what really makes a difference in psychotherapy outcomes Duncan, Miller, Wampold & Hubble (Eds.), Heart & Soul of Change (2 nd ed.). Washington, DC: American Psychological Association

6. 2.What is “Outcomes Informed Care?” Outcomes Informed Care is the philosophy and method of providing treatment and care that is informed by outcomes reported by patients.

7. 2.What is “Outcomes Informed Care?” Outcomes Informed Care is done by: Collecting the patient’s clinical symptoms and perception of therapeutic alliance throughout the course of treatment using questionnaires. Questionnaires are administered frequently so as to actively inform the patient’s course of treatment. Clinicians are provided continuous feedback on these outcomes to actively improve treatment.

8. 3.Background Research a.The “Dodo Bird Effect” b.Clinician effects c.The relationship between the patient and clinician d.Clinician’s use of feedback e.Outcomes benchmarking

9. The “Dodo Bird Effect” Taken from the article by Rosenzweig S. (1936) titled, “Some implicit common factors in diverse methods of psychotherapy: At last the Dodo said, ‘Everybody has won and all must have prizes.’” (Am J Orthopsychiatry 6:412-5) Refers to the conjecture that all psychotherapies intended to be effective are roughly equivalent in their treatment outcomes due to factors that are common to all psychotherapies 3.Background Research a. The “Dodo Bird Effect”

10. Common factors Three decades of meta-analytic studies have served to reinforce Rosenzweig’s (1936) observation (e.g., Wampold et al., 1997) Lack of evidence for specific treatment effects bolster the argument that almost all of the effects of psychotherapy are due to factors common to all psychotherapies No evidence that effects of treatments have increased over the past three decades in psychotherapy and pharmacotherapy research 3.Background Research a. The “Dodo Bird Effect”

11. Meta-analyses of psychotherapy outcomes reveal that the treatment effect due to the clinician are larger than the techniques used in the treatment Recent analyses show that who prescribes the medication also affects treatment outcomes in pharmacotherapy outcomes Other recent analyses suggest that the psychotherapist even effects the effect of medication in treatments that combine both psychotherapy and pharmacotherapy! 3.Background Research b. Clinician Effects

12. Relationship building is an Evidence Based Practice! “Practitioners are encouraged to routinely monitor patients’ responses to the therapy relationship and ongoing treatment. Such monitoring leads to increased opportunities to repair alliance ruptures, improve the relationship, modify technical strategies, and avoid premature termination.” —John Norcross & Michael Lambert (2006) from “The Therapy Relationship” in Norcross, Beutler, & Levant (Eds.), Evidence-Based Practices in Mental Health, p. 218 3.Background Research c. Relationship Between The Patient and Clinician

13. Practice implications for relationship building Listen to client Privilege the client’s experience Request feedback on the therapy relationship Avoid critical or pejorative comments Ask what has been most helpful in this therapy —John Norcross (2009) from “The Therapeutic Relationship” in Duncan, Miller, Wampold & Hubble (Eds.), Heart & Soul of Change (2 nd ed.), pp. 116-117 3.Background Research c. Relationship Between The Patient and Clinician

14. Research indicates that routine use of questionnaires provides clinicians ability to identify “at risk” cases and prevent premature termination Statistical algorithms compare actual patient improvement against “expected” improvement based on large normative samples and provide this information to clinicians Strong evidence from controlled studies and real world applications that patients benefit 3.Background Research d. Clinician’s Use of Feedback

15. “Yes, it is time for clinicians to routinely monitor treatment outcome” “The use of outcomes management systems is ushering in a significant change in how psychotherapy is conducted. This review underscores the value of monitoring treatment response, applying statistical algorithms for identifying problematic cases, providing timely feedback to therapists (and clients), and providing therapists with problem- solving strategies. It is becoming clear that such procedures are well substantiated, not just matters for debate or equivocation. When implemented, these procedures enhance client outcome and improve quality of care.” —Michael Lambert (2009) from “Yes It Is Time for Clinicians to Routinely Monitor Treatment Outcomes” in Duncan, Miller, Wampold & Hubble (Eds.), Heart & Soul of Change (2 nd ed.), p. 259. 3.Background Research d. Clinician’s Use of Feedback

16. Benchmarking refers to the practice of comparing one set of outcomes to a criterion (the benchmark) Commonly, comparison of effect sizes to “benchmarks” obtained from meta-analyses of clinical trials An individual clinician’s average effect size could also be compared against effect sizes of other clinicians in a large normative sample 3.Background Research e. Outcomes Benchmarking

17. 4.Measurement Concepts a.Measurement 2.0 b.Questionnaire development i.Global distress factor ii.Therapeutic alliance iii.Reliability & validity iv.Item response theory c.Effect size d.Analysis i.Case mix adjustment ii.Inadequacies of analyses used in clinical trials iii.Hierarchical linear modeling

18. Measurement 1.0 Reliance on copyrighted and published questionnaires Copyright holder may charge fees for the use of questionnaires Copyright holder may place conditions or restrictions on the use of questionnaires Measurement 2.0 Item banks and questionnaires that use them belong to the community of users No fees for questionnaires constructed from items in the shared item bank Each organization is responsible for determining the appropriate content (items) and implementation of questionnaires General differences: 4.Measurement Concepts a. Measurement 2.0

19. Measurement 1.0 A pool of items are tested in various samples Item analysis used to select items for final questionnaire Questionnaire is usually validated through factor analysis and studies correlating it with other questionnaires measuring the same construct Questionnaire published in final and unchangeable form Manual for questionnaire published in paper form Many years may pass before a new version is published Measurement 2.0 A pool of items are tested in various samples Item analysis used to select items; multiple versions of the questionnaires are created by combining a set of items that reflect the needs of the users Questionnaire validated by factor analysis and correlational studies Questionnaires are constantly updated as data accumulate and needs of users change Online manual constantly updated as data accumulate and needs of users change New version is immediately available if the other versions do not capture the unique needs of the users Differences in questionnaire development: 4.Measurement Concepts a. Measurement 2.0

20. 4.Measurement Concepts b. Questionnaire Development i.What is “global distress?” Most measures commonly used in mental health research correlate strongly with a common factor, commonly called the “global distress factor”“global distress factor” Global distress includes items measuring: Symptoms of depression and anxiety Attention and concentration problems Family and interpersonal relations Work place productivity and functionality

21. 4.Measurement Concepts b. Questionnaire Development ii.Therapeutic alliance Measuring the quality of the therapeutic relationship between the patient and the clinician during the session Three Components: Task: Behaviors and processes within the therapy session that constitute the actual work of therapy Bond: The positive interpersonal attachment between therapist and client of mutual trust, confidence, and acceptance Goal: Objectives of therapy that both client and therapist endorse

22. 4.Measurement Concepts b. Questionnaire Development ii.Therapeutic alliance Sample items: I felt like we talked about the things that were important to me I felt like the therapist liked and understood me I felt the session was helpful I felt confident that the therapist and I worked well together Did you feel that the clinician understood what it was like to be you?

23. 4.Measurement Concepts b. Questionnaire Development iii.Reliability and validity Reliability It asks, “how consistent is this questionnaire?” Cronbach’s coefficient alpha ( α ) estimates how consistent the items within this questionnaire is α =.9 or higher preferred for measures Validity It asks, “does the questionnaire measure what it is supposed to measure?” Measures of global distress will correlate highly with other similar measures (construct validity) Items are representative of what is supposed to be measured (content validity)

24. 4.Measurement Concepts b. Questionnaire Development iv.Item Response Theory (IRT) Evaluates how clients at varying levels of distress reply to each items Identifies items that are endorsed by clients with high levels of distress E.g., thoughts of suicide; worthlessness Questionnaires can be uniquely tailored to the users or setting Questionnaires for outpatient treatments are appropriately calibrated for patients with moderate to severe symptoms

25. Sample ACORN Questionnaire: 4.Measurement Concepts b. Questionnaire Development Global distress Suicidal Ideation Substance abuse Productivity Global distress Alliance Questionnaires faxed to data center for cost effective scoring, analysis & data warehousing

26. What is effect size? Effect size is simply the magnitude of the treatment effect Of many effect size units, Cohen’s d is very informative for treatments An effect size of d= 1 means that the client improved one standard deviation on the outcome measure On a global distress scale, an effect size of d= 0.8 or higher is considered large Meta-analyses of large samples of psychotherapy studies suggest the effect size for psychotherapy using a global distress scale is approximately d= 0.8 4.Measurement Concepts c. Effect Size

27. 4.Measurement Concepts d. Analysis i.Case mix adjustment Statistical adjustments are used to control for case mix differences (severity adjusted effect size) General linear model (GLM) is commonly used GLM use both categorical (e.g., diagnosis) and continuous (e.g., severity) variables in predictive model Predictive model employs clinically relevant variables collected at the start of treatment to predict improvement during treatment Initial severity, as measured by the intake score, is the strongest predictor Other predictors: diagnosis, treatment history, health

28. 4.Measurement Concepts d. Analysis ii.Inadequacies of analyses used in clinical trials Clinical trials typically: Randomly assign patients to treatment conditions Conduct analysis of variance (ANOVA) to compare among different treatment conditions to see whether or not treatment results are “statistically significant” Analysis assumes that the clinicians are all the same ANOVA as commonly employed in clinical trials cannot be used in assessing real-world treatment effects because there are clear differences among clinicians in their average treatment outcome

29. 4.Measurement Concepts d. Analysis iii.Hierarchical linear modeling (HLM) Clinicians are treated as a factor that influences outcomes Patients are “nested” within clinician Estimates percentage of variance due to clinician Use of HLM becomes a necessity because we cannot ignore the effects due to the individual clinicians Ignoring effects due to clinicians can lead to erroneously concluding that the type of treatment makes a difference in outcomes

30. 5.Clinician’s Use of Feedback Monitoring trajectory of change: Graphing change permits clinician to quickly identify cases that are “off track” Expected change based on normative sample Signal score: patient “off track” and at risk for poor outcome Patient’s actual scores: probability of good outcome remains high if patient in the “clinical” range remains engaged in treatment. Clinical Boundary: Scores above this line are in a clinical range

31. 5.Clinician’s Use of Feedback Monitoring effect size: Data warehouse and web based tool permits health plans and managed care companies to monitor effect size Many health plans and managed care companies permit clinicians to access their data and monitor their own outcomes Outcomes informed clinicians can use data to market their services

32. 5.Clinician’s Use of Feedback Monitoring effect size: Severity Adjusted Effect Size:  d = 0.5~0.8 (effective)  d > 0.8 (highly effective)

33. 5.Clinician’s Use of Feedback Therapeutic alliance scale Items are heavily skewed in positive direction I.e., clients report “perfect” or “near perfect” therapeutic alliance with their therapist Scale scores are not normally distributed Cannot calculate reliability & validity using parametric statistics that assume normality of distribution Items are only as “valid” as clinician’s ability to illicit honest and frank responses!

34. 5.Clinician’s Use of Feedback Therapeutic alliance scale results Alliance analyses performed by Jeb Brown, PhD using data from the ACORN data repository

35. 5.Clinician’s Use of Feedback Therapeutic alliance scale results Measuring alliance clearly makes a difference

36. 6.Financial Implications Clinicians vary in their “value” Value Index estimates effect size per $1000 Value Index Effective clinicians produce greater “return on investment” of employers Increased productivity gains for treated employees Reduced medical costs Effective clinicians enhance the “value” of medications Clinicians with good outcome data could have competitive advantage in attracting business

37. 6.Financial Implications Certificate of Effectiveness Similar to concept of “Certified Organic” Process Data analyzed by independent party Applies agreed upon criteria, including minimum effect size and sample size Purpose Increase customer confidence of “value” Enable clinicians to demonstrate effectiveness to referral sources such as employers, health plans, and HMOs Empower clinicians to compete for business and negotiate contract based on demonstrated “value”

39. Psychotherapy Works. It Works! “Monitoring combined with feedback is a simple method, divorced of theoretical baggage, for providing accountability. The results are apparent to all who have an interest in the outcome: therapists, consumers, administrators, and payers. Accepting the premise that therapeutic factors constitute the engine of change, then monitoring and feedback offers the means to deliver them. Many are anxious about the future and deservedly so. At the same time, the profession has the opportunity to establish itself in its own right. Psychotherapy works. It works. Therapists now have the ability to show it and the means to banish the despair in the workforce. The challenge is to put it into practice.” —Barry Duncan, Scott Miller, Bruce Wampold, & Mark Hubble (2009) from “Introduction” in Duncan, Miller, Wampold & Hubble (Eds.), Heart & Soul of Change (2 nd ed.), p. 40