1. Understanding Cutoffs, Norms, Trajectories, and everything else that freaks you out: A Primer for Dummies G.S. (Jeb) Brown, Ph.D. Center for Clinical Informatics

2. But I’m a clinician…… Statistics aren’t necessary to be a good therapist. I was never good at math. I’ll let the researchers worry about statistics, I’m interested in helping clients. I know when my clients are getting better; I don’t need an outcome measure to tell me this. I dissociate when confronted with numbers.

3. Stats phobia desensitization Close you eyes and breath slowly Visualize a lovely hill… green grass, gentle breeze, warm sun See how the grand rises slowly at first, then steeper; notice the sensuous lines and beautifully rounded summit. As you exhale slowly, say to yourself … “I love the Bell Curve”. Repeat as needed.

4. Mantra: I love the Bell Curve

6. What’s normal? All outcome questionnaires appear to measure a common “factor”… global distress/happiness Like almost every other human trait, misery is normally distributed.

7. Calculating clinical cutoff Clinical cutoff scores used to estimate the boundary between “normal” and “clinical”. Take another deep breath… here comes a formula…… Thanks to Jacobson & Truax, 1991

8. Idealized clinical cutoff

9. Nothing is perfect…. The Bell Curve is beautiful in the abstract, but reality is messier. Many tests may floor and ceiling effects. Distributions may depart from normality….

10. Floor and ceiling effects Example – the mean of a in a non clinical sample completing the ORS was 30 and the standard deviation 6.2. But wait! The ORS has a maximum of score of 40, only 1.6 standard deviations above the mean.

11. Example samples (sample samples?) Accountable Behavioral Healthcare Alliance –Five county community mental health system in Oregon serving adults and children. ORS administered administered at every session. Data collected at an estimated 80% of all sessions. RFL-Resources for Living –EAP company providing telephonic counseling services to adults. ORS administered telephonically at every session. Approximately 80% of clients have a single session (one call). SAIC –Clinics in multiple countries serving children of active duty US military personnel and military contractors. Community sample- Small sample of non clients.

12. ORS score distributions 4 clinical & 1 community samples

13. Non normal curves……

14. Clinical cutoff scores

15. When is change “real” All measurement has “error” How do we know if change on a test is simply the result of random error? The Reliable Change Index is a common metric to determine if the difference between two scores is greater than expected from random error. Here we go again….Jacobson & Truax, 1991

16. When is change “real” Standard error of measurement (S E ) is defined as the standard deviation multiplied by he square root of 1 minus the reliability of test as calculated by the coefficient alpha, a measure of internal consistency. Estimates of coefficient alpha…. ABHA =.89 RFL=.82 SAIC=.97

17. RCI estimates Did I mention that all measurement is approximation?

18. Criteria for recovery Jacobson & Truax (1991) proposed a two fold criteria for recovery 1.Change score exceeds the RCI 2.Scores moves from clinical range to normal range. Two problems with this criteria… 1.A substantial percentage of patients start treatment in the normal range 2.The probability of change exceeding the RCI is a function of severity

19. % of cases with “real” change (Change exceeds 5 points) All cases Clinical range cases (ORS intake =< 25)

20. Case mix adjustment We can’t evaluate outcomes without answering the question “Compared to what?” Differences in severity (intake score) and other client characteristics make it difficult to compare outcomes Case mix adjustment uses statistical methods to adjust for differences in clients when comparing outcomes from one site to another Case mix adjustment is always imperfect.

21. Regression to the rescue

22. Regression artifacts are always present in repeated measures If a measure has test-retest reliability (correlations between two points in time) then it will exhibit regression to the mean. Correlations between measures will tend to decrease over time. The intake score is almost always the strongest predictor of change. Recommended reading – A Primer on Regression Artifacts (Campbell & Kenny, 1999)

23. Multivariable GLM The General Linear Model can incorporate multiple variables to predict a continuous dependent variable, such as a change score. Both continuous and categorical variables can be incorporated into the model. Multivariate analysis of variance using GLM suggest that most of the variance in change scores is explained by the intake score. Variables such as diagnosis, age and sex account for a relatively small percentage of the variance.

24. Multivariable Regression

25. Excel and regression The slope and intercept values for a simple linear regression can be calculated in Excel. Amazingly, the slope function returns the coefficient for the slope and the intercept function returns the value for the intercept. Different slopes and intercepts can be calculated for subgroups (age group, diagnosis, etc).

26. Benchmarking outcomes GLM permits “benchmarking” of outcomes, by comparing an actual score (or change score) with the predicted score derived some the comparison (or normative) sample. Difference between actual and predicted score is know as the residual score (change score) “Benchmark Score” or Change Index Score” maybe have more intuitive meaning than “Residual Score”. Or not.

27. Trajectory of Change Graph The Trajectory of Change graphs displays a client’s actual scores to the predicted scores. GLM used to predict scores are subsequent measurement intervals (sessions, weeks) using intake score and any other variables available. Distribution of residuals used to plot percentiles at different intervals.

28. Graphing scores over time

29. Graphing 3 way interactions Trajectory of Change graphs can be used to display 3 way interactions involving severity, time and a third variable of interest (treatment, age group, diagnosis, etc.) Slope and intercept at each measurement point are calculated separately for grouping of interest. Following are two examples from the ABHA data.

30. Trajectories of change Adults and Children/adolescents (AHBA)

31. Trajectories of change Before and after feedback (AHBA)

32. Norms and Benchmarks Regression formulas can be used to create norms for change. Outcomes can be “Benchmarked” by determining the difference between the expected change (using the regression formula) and the actual change By contributing data to common data repository, ORS users can assure that norms are continuously refined and updated.

33. References 1.Science Cartoons by Georg Meixner http://www.vias.org/science_cartoons/ 2.Jacobson NS & Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. 1991;59:12-19. 3.Campbell, DT & Kenny, DA. A Primer on Regression Artifacts. The Guildford Press, 1999.

34. About the presenter G.S. (Jeb) Brown is a licensed psychologist with a Ph.D. from Duke University. He served as the Executive Director of the Center for Family Development from 1982 to 19987. He then joined United Behavioral Systems (an United Health Care subsidiary) as the Executive Director for of Utah, a position he held for almost six years. In 1993 he accepted a position as the Corporate Clinical Director for Human Affairs International (HAI), at that time one of the largest managed behavioral healthcare companies in the country. In 1998 he left HAI to found the Center for Clinical Informatics, a consulting firm specializing in helping large organizations implement outcomes management systems. Client organizations include PacifiCare Behavioral Health/ United Behavioral Health, Department of Mental Health for the District of Columbia, Accountable Behavioral Health Care Alliance, Resources for Living and assorted treatment programs and centers throughout the world. Dr. Brown continues to work as a part time psychotherapist at behavioral health clinic in Salt Lake City, Utah. He does measure his outcomes.