1. Kaiser Permanente Outcomes Laboratory G.S. (Jeb) Brown, Ph.D. Center for Clinical Informatics

2. What is outcomes informed care? Frequent administration of patient self report questionnaires in order to monitor patient response to treatment Use of decision support tools to inform clinical judgment Analysis of outcomes data to determine sources of variance: practice based evidence rather than evidence based practice Use of practice based evidence to identify pathways to improved outcomes and to monitor success

3. Why measure outcomes? A large, growing body of research over the past decade suggests that routine measurement of outcomes leads to improved outcomes, particularly for those patients most at risk. 1-13 Decades of research support the assertion that different methods of psychotherapy produce similar results. 14-19 (cont.)

4. Why measure outcomes (cont.) –Hierarchical Linear Modeling reveals that the clinician is more powerful than the technique in the variance in outcomes. 20-32 –Some recent analyses show the prescribing clinician is at least as powerful as the drug in pharmacotherapy. –So, in behavioral healthcare, focusing on the treatment (therapy technique, medication) is not enough to optimize outcomes – we must also focus on the outcomes-informed clinician.

5. An outcomes-informed clinician… Uses the best available data on treatment outcome to inform the treatment for each client/patient Stays current on the latest research on what makes a difference in treatment outcomes. Recognizes the importance of clinician skill in providing effective treatments. Supports the desire to improve outcomes by actively evaluating them, and applying the feedback to the treatment.

6. www.psychoutcomes.org Non-profit set up to encourage the use of client/patient-completed outcome measures in behavioral health care and related fields. TWiki site provides information, fosters collaboration, and offers support for organizations launching and nurturing outcomes-informed care initiatives.

8. KP Outcomes Lab http://www.psychoutcomes.org/KP Secure web for use by Kaiser Permanente staff ACORN user name and password required Links to download questionnaires and view outcome data Links to online questionnaire manuals, research results, and knowledge base on outcomes measurement methodology

10. Items, not questionnaires KP pilot project takes advantage of the ACORN item inventory.ACORN item inventory. Items have been field tested in large community and clinical samples. Items are selected from the inventory for inclusion on a form based on the specific measurement needs of each organization. Allows for much greater flexibility and “elegance” than reliance on copyrighted questionnaires with a fixed number of items.

12. Regression implications Patients with high distress report greater overall change and greater change per session than low distress patients. Patients with scores in ‘normal’ (non-clinical) range tend to report little improvement or even show increased distress over time. Focusing treatment resources on patients with the most severe symptoms results in improved outcomes.

13. Benchmarking outcomes Measuring outcomes is of little use without some basis of comparison. Are the outcomes good? Compared to what? Clinicians and organizations differ in the kinds of cases they treat. Benchmarking outcomes requires a method of accounting for differences in case mix.

14. GLM and Case Mix Adjustment Case mix variables are those variables present at the beginning of the treatment episode that make a difference in the outcome (they help predict). General Linear Modeling uses categorical variables such as age group, sex and diagnosis, combined with intake scores, to predict a future score. The difference between the predicted score and the patient’s actual score at the future measurement point is referred to as the “residual score.”

15. Predicting change “Trajectory of change” graphs are created for each patient using GLM to predict scores at future measurement points. Actual patient scores are plotted against the predicted scores. Patients whose actual scores are more than a standard deviation worse than the predicted score are targeted as “Signal cases;” they are at risk for early termination and a very poor outcome. These patients are likely to improve if they remain engaged in treatment.

17. Benchmarks: Target & Score ACORN reports use the term “Benchmark Target” to refer to the predicted final score in a treatment episode. This Target is calculated using all episodes in the data base, regardless of number of sessions. “Benchmark Score” is the residual score – i.e., the difference between that Target and the actual score in the treatment. Positive Benchmark scores are great; significantly negative Benchmark scores are signal alerts.

18. Signal case Positive scores – great!

19. Decision Support Toolkit The ACORN Decision Support Toolkit is an Excel- based report which makes use of built-in macros to help the clinician view and graph outcome data for each patient. Graphing utility includes graphs for subscales, predicted change, Benchmark Target and Signal line. Summary statistics include mean change and Benchmark (residual) scores.

20. Clinician’s Desktop Each clinician has a personal secure Clinician’s Desktop web page with access to data on his or her patients. Clinician’s Desktop includes:  Links to download questionnaires  Links to data files (Decision Support Toolkits)  Frequently Asked Questions To get to the Clinicians’ Desktop the first time...

21. Click on the KP link

22. Click on your site

23. Click on your name

24. Voila! To see your data, click on your Decision Support Toolkit

25. Once you have gone through all these steps, and found your personal webpage the first time, you can make this all much quicker by creating a shortcut to your page on your computer’s desktop, as follows….

26. Left-click on this “e” icon, and drag to your desktop…

27. Now you have a shortcut to your personal webpage right on your Desktop! Click here whenever you want to log in and see your data.

28. How to use graphs… Use the Toolkit screen to identify the patient you want to graph. Place curser on number of the row containing the patient data; click to highlight the entire row. Click on “View Client Graph” Trajectory of Change Graph screen includes buttons for graphing subscales, Projected Change, Benchmark Target and Signal scores

29. 1. Click here to highlight the case… 2…then click here.

31. References 1.Sapyta J, Riemer M, Bickman L. 2005. Feedback to therapist: theory, research, and practice. J Clin Psychol 61(2):145-53.Sapyta J, Riemer M, Bickman L. 2005. 2.Hannan C, Lambert MJ, Harmon C et al. 2005. A lab test and algorithms for identifying clients at risk for treatment failure. J Clin Psychol 61(2):155-63.Hannan C, Lambert MJ, Harmon C et al. 2005 3.Lambert MJ, Harmon C, Slade K et al. 2005. Providing feedback to psychotherapists on their patients progress: Clinical results and practice suggestions J Clin Psychol 61(2):165-74.Lambert MJ, Harmon C, Slade K et al. 2005 4.Harmon C, Hawkins, Lambert MJ et al. 2005. Improving outcomes for poorly responding clients: The use of clinical support tools and feedback to clients. J Clin Psychol 61(2):175-85.Harmon C, Hawkins, Lambert MJ et al. 2005 5.Brown GS, Jones ER. 2005. Implementation of a feedback system in a managed care environment: What are patients teaching us? J Clin Psychol 61(2):187-98.Brown GS, Jones ER. 2005. 6.Miller SD, Duncan BL, Ryan S, et al. 2005. The Partners for Change Outcome Management System. J Clin Psychol 61(2):199-208.Miller SD, Duncan BL, Ryan S, et al. 2005.

32. References (continued) 7.Claiborn CD, Goodyear EK. 2005. Feedback in psychotherapy. J Clin Psychol 61(2):209-21.Claiborn CD, Goodyear EK. 2005. 8.Brown GS, Burlingame GM, Lambert MJ, et al. 2001. Pushing the quality envelope: A new outcomes management system. Psychiatr Serv 52(7):925-34.Brown GS, Burlingame GM, Lambert MJ, et al. 2001. 9. Lueger RJ. 1998. Using feedback on patient progress to predict the outcome of psychotherapy. J Clin Psychol 54:383-93.Lueger RJ. 1998. 10. Lambert MJ, Whipple JL, Smart DW, et al. 2001. The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychother Res 11(1):49-68.Lambert MJ, Whipple JL, Smart DW, et al. 11.Lambert MJ, Whipple JL, Vermeersch DA, et al. 2002. Enhancing psychotherapy outcomes via providing feedback on client progress: A replication. Clin Psychol Psychother 9:91-103.Lambert MJ, Whipple JL, Vermeersch DA, et al.

33. References (continued) 12.Whipple JL, Lambert MJ, Vermeersch DA, et al. 2003. Improving the effects of psychotherapy: The use of early identification of treatment failure and problem-solving strategies in routine practice. J Counsel Psychol 50(1):59-68.Whipple JL, Lambert MJ, Vermeersch DA, et al. 13.Lambert MJ, Whipple JL, Hawkins EJ, et al. 2003. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clin Psychol Sci Prac 10:288-301. 14.Rosenzweig S. 1936. Some implicit common factors in diverse methods of psychotherapy: “At last the Dodo said, ‘Everybody has won and all must have prizes.’” Am J Orthopsychiatry 6:412-5. 15.Shapiro DA, Shapiro D. 1982. Meta-analysis of comparative therapy outcome studies: A replication and refinement. Psychol Bull 92:581-604.

34. References (continued) 16.Robinson LA, Berman JS, Neimeyer RA. 1990. Psychotherapy for treatment of depression: A comprehensive review of controlled outcome research. Psychol Bull 108:30-49.Robinson LA, Berman JS, Neimeyer RA. 1990. 17.Wampold BE, Mondin GW, Moody M, et al. 1997. A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychol Bull 122:203-15. 18.Ahn H, Wampold BE. 2001. Where oh where are the specific ingredients? A meta-analysis of component studies in counseling and psychotherapy. J Counsel Psychol 48:251-7. 19.Wampold BE. 2001. The great psychotherapy debate: Models, Methods, and Findings. Mahwah NJ: Lawrence Erlbaum Associates. 272 pp.Wampold BE

35. References (continued) 20.Martindale C. 1978. The therapist-as-fixed-effect fallacy in psychotherapy research. J Consult Clin Psychol 46:1526-30. 21.Luborsky L, Crits-Christoph P, McLellan T, et al. 1986. Do therapists vary much in their success? Findings from four outcome studies. Am J Orthopsychiatry 56:501-12. 22.Crits-Christoph P, Baranackie K, Kurcias JS, et al. 1991. Meta-analysis of therapist effects in psychotherapy outcome studies. Psychother Res 1:81- 91. 23.Crits-Christoph P, Mintz J. 1991. Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. J Consul Clin Psychol 59:20-6.Crits-Christoph P, Mintz J. 1991. 24.Wampold BE. 1997. Methodological problems in identifying efficacious psychotherapies. Psychother Res 7:21-43,

36. References (continued) 25.Elkin I. 1999. A major dilemma in psychotherapy outcome research: Disentangling therapists from therapies. Clin Psychol Sci Prac 6:10- 32. 26.Wampold BE, Serlin RC. 2000. The consequences of ignoring a nested factor on measures of effect size in analysis of variance designs. Psychol Methods 4:425-33. 27.Huppert JD, Bufka LF, Barlow DH, et al. 2001. Therapists, therapist variables, and cognitive-behavioral therapy outcomes in a multicenter trial for panic disorder. J Consul Clin Psychol 69:747-55.Huppert JD, Bufka LF, Barlow DH, et al. 2001. 28.Luborsky L, Rosenthal R, Diguer L, et al. 2002. The dodo bird verdict is alive and well—mostly. Clin Psychol Sci Prac 9:2-12. 29.Okiishi J, Lambert MJ, Nielsen SL, et al. 2003. Waiting for supershrink: An empirical analysis of therapist effects. Clin Psychol Psychother 10:361-73.

37. References (continued) 30.Brown GS, Jones ER, Lambert MJ, et al. 2005. Identifying highly effective psychotherapists in a managed care environment. Am J Managed Care 11(8):513-20.Brown GS, Jones ER, Lambert MJ, et al. 2005. 31.Wampold BE, Brown GS. 2005. Estimating therapist variability: A naturalistic study of outcomes in private practice. J Consul Clin Psychol.73(5): 914-923.Wampold BE, Brown GS. 32.Kim DM, Wampold BE, Bolt DM. 2006. Therapist effects and treatment effects in psychotherapy: Analysis of the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychother Res. 16(2):161-172.

38. About the presenter G.S. (Jeb) Brown is a licensed psychologist with a Ph.D. from Duke University. He served as the Executive Director of the Center for Family Development from 1982 to 1987. He then joined United Behavioral Systems (a United Health Care subsidiary) as the Executive Director for Utah, a position he held for almost six years. In 1993 he accepted a position as the Corporate Clinical Director for Human Affairs International (HAI), at that time one of the largest managed behavioral healthcare companies in the country. In 1998 he left HAI to found the Center for Clinical Informatics, a consulting firm specializing in helping large organizations implement outcomes management systems. Client organizations include Resources for Living, Regence, United Behavioral Health, Accountable Behavioral Health Care Alliance, and assorted treatment centers. Dr. Brown continues to work as a part time psychotherapist at a behavioral health clinic in Salt Lake City, Utah. He does measure his outcomes.