Assessment of Reliable Change: Methods and Assumptions Michael Basso, Ph.D. Associate Professor and Director of Clinical Training Department of Psychology—University of Tulsa Clinical Associate Professor Department of Psychiatry—University of Oklahoma

Objectives Provide background concerning methods of assessing reliable change Describe assumptions and applications of reliable change scores Illustrate use of reliable change scores

Assessment of Clinical Change Two Basic Approaches –Assessment of Group Differences Across Time –Assessment of Individual Differences Across Time

Assessment of Group Differences Across Time Assessment of statistically reliable change –“Does the treatment yield significant benefits for groups of patients?” »i.e., do average scores at T1 and T2 come from different distributions –This approach describes the average rate of change over groups primarily –It is accomplished with repeated measure ANOVA –Problem: You could have a statistically significant difference with a very small effect size, but it might not be a clinically meaningful change

Assessment of Clinically Meaningful Change –“Did the patient’s change in performance at T2 exceed base rates of change?” »i.e., did the individual show change that exceeded expectations based on measurement error, practice effects, and regression to the mean? –This method describes the base rate of change »Change that is exceeds the base rate is not normal, and is therefore clinically meaningful –Our focus is on the assessment of clinically meaningful change in individuals, but this method can be applied to group data as well Assessment of Individual Differences Across Time

How do you establish the base-rate of change? Bear into consideration that: –It would be improbable to obtain the exact same score twice –There is no perfect test-retest correspondence because of »measurement error »regression to the mean »practice effects Assessment of Clinical Change for Individuals

Reliable change Index scores –“Does change exceed what would be expected based on measurement error alone?” –This method is based on Reliability of measurement –It is used for typical performance tests »i.e., attitude, personality, psychopathology, etc. Standardized Regression-Based Change Scores –“Does change in scores exceed expectations based on T1 (baseline) scores?” –This method is based on a validity coefficient (i.e., what T2 score is predicted by the T1 score) –It is used for maximal performance tests »i.e., IQ, neuropsychological, etc. Two Methods of Assessing Base Rates of Change

Elaborated by Jacobson and Truax (1991) –Based on the standard error of the difference »Which in turn is based on the reliability coefficient –This reflects the sampling distribution of difference scores –it implies the magnitude of differences between two test scores that vary by chance alone Assumptions –Error components are mutually independent and independent of true pretest and posttest scores –Error is normally distributed with a mean of 0 –SE of error is equal for all subjects –These assumptions are questionable in clinical settings (cf. Maassen, 2004) Reliable Change Index Scores

Standard Normal Curve—Distribution of Difference Scores

To use the RCI, you must compute the SE of difference between two scores SEdiff=(2(SD(1-r xx ) 1/2 ) 2 ) 1/2 Then, compute a confidence interval for change scores for 95% confidence, you multiply 1.96 * SEdiff for 90% confidence, you multiply 1.60 * Sediff Does the raw score change between T2 and T1 exceed the confidence interval? –If so, it represents change that exceeds the base rate expected based on measurement error »Thus, clinically meaningful change has occurred –If not, then the change is consistent with the base rate expected based on measurement error »Thus, no clinically meaningful change has occurred Reliable Change Index Scores--Method

Ferguson, Robinson, & Splaine (2002) –SF-36 in 200 patients who had undergone a Coronary Artery Bypass Grafting (CABG) surgery –SF-36 contains 8 scales »Physical Functioning »Role Functioning Physical »Bodily Pain »General Health »Vitality »Social Functioning »Role Functioning-Emotional »Mental Health Reliable Change Index Scores—An Example

Ferguson, Robinson, & Splaine (2002) –Physical Functioning »Reliability=.93 (from normative sample of 2474) »Mean of normative sample=84.15 »SD of normative sample=23.28 »SEdiff=(2(SD(1-r xx ) 1/2 ) 2 ) 1/2 SEdiff=(2(23.28(1-.93) 1/2 ) 2 ) ) 1/2 = % CI: (SEdiff)*1.96=19.32 T1 Mean=40.97 T2 Mean=63.39 Mean Diff=22.42 The mean difference exceeds Thus, clinically meaningful change has occurred as a result of surgery Reliable Change Index Scores—An Example

Ferguson, Robinson, & Splaine (2002) –Mental Health »Reliability=..84 (from normative sample of 2474) »Mean of normative sample=75.01 »SD of normative sample=21.40 »SEdiff=(2(SD(1-r xx ) 1/2 ) 2 ) 1/2 SEdiff=(2(21.40(1-..84) 1/2 ) 2 ) ) 1/2 = % CI: (SEdiff)*1.96=21.40 T1 Mean=72.08 T2 Mean=71.84 Mean Diff=-0.24 The mean difference fails to exceed Thus, no clinically meaningful change has occurred as a result of surgery Reliable Change Index Scores—An Example

Elaborated by Charter (1996) Used primarily for maximal performance tests The RCI of Jacobsen and Truax is used for typical performance tests –It assumes that errors between test scores at baseline and time 2 are uncorrelated –This assumption is untenable in maximal performance tests because of practice effects Standardized Regression Based Change Scores

Based on the standard error of prediction –SE pred =SD Y2 ((1-r Y1Y2 2 ) 1/2 ) The SE reflects the sampling distribution of predicted scores It implies the range of scores that might be expected at time two that may be expected from the baseline score and prediction error This method requires you to compute the estimated true score –Y2 True =M+((r Y1Y2 )(Y1-M)) The T2 score is prone to error, and this formula permits an unbiased estimate of the true score The SEpred is used to compute a confidence interval around the estimated true score Standardized Regression Based Change Scores

Standard Normal Curve—Distribution of Standard Error of Prediction Around Estimated True Score

To use the SRB, you must compute the estimated true T2 score Compute the confidence interval around this estimated true T2 score For 95% confidence, you multiply 1.96 * SEpred For 90% confidence, you multiply 1.60 * SEpred Does the obtained T2 score fall outside the confidence interval around the estimated true score for T2? –If so, it represents change that exceeds the base rate expected based on measurement error, regression to the mean, and practice »Thus, clinically meaningful change has occurred –If not, then the change is consistent with the base rate expected based on measurement error, practice, and regression to the mean »Thus, no clinically meaningful change has occurred Standardized Regression Based Change Scores--Method

Basso, Carona, Lowery, & Axelrod (2002) –WAIS-III re-tested in a group of control subjects over a 3-6 month interval –FSIQ »Test-Retest Reliability=.90 »T1 Mean T1=109.4 (11.6) »T2 Mean T2=115.0 (12.1) »SE pred =SD Y2 ((1-r Y1Y2 2 ) 1/2 ) SE pred =(12.1(( ) 1/2 ))= % CI: (SEdiff)*1.96=10.36 Mean Diff=5.60 –The mean difference fails to exceed the 95% CI –No individual had a score exceeding the 95% CI To apply the SRB, the T2 True Score is estimated –If the obtained score falls within the CI around the T2 True score, then no clinically meaningful change has occurred Standardized Regression Based Change Scores--An Example

Basso, Carona, Lowery, & Axelrod (2002) –An example application: –T1 obtained score=104 –T2 obtained score=116 –Estimated True T2 Score »Y True =M+((r Y1Y2 )(Y1-M)) »Y True =100+(.90)( )=103.6 »116 exceeds points from »Thus, meaningful change has occurred Standardized Regression Based Change Scores--An Example

Basso, Carona, Lowery, & Axelrod (2002) –An example application: –T1 obtained score=103 –T2 obtained score=106 –Estimated True T2 Score »Y True =M+((r Y1Y2 )(Y1-M)) »Y True =100+(.90)( )=105.4 »105 falls within points of 106 »Thus, no meaningful change has occurred Standardized Regression Based Change Scores--An Example

Questions?