Analysis of Covariance vs. Analysis of Difference Scores Lord’s Paradox Analysis of Covariance vs. Analysis of Difference Scores Lord, F. M. (1967). A paradox in the interpretation of group comparisons. Psychological Bulletin, 68 (5). 304-305.

Inconsistency in significance testing In analyzing how subjects change, analysis of covariance, and analysis of difference scores yield different results The same data sets, analyzed for the same effects, using different techniques yielded qualitatively different results

Lord’s example A test of sex differences in the “Freshman 15” Analysis of difference scores for males and females shows no difference Analysis of covariance shows a significant difference So, what is going on?

-Distributions on top = initial weight distributions -Distributions on side = final weight distributions -Ovals = scatter plots of male and female weights

“with the data usually available for such studies, there simply is no logical or statistical procedure that can be counted on to make proper allowances for uncontrolled pre-existing conditions between groups” -Frederic Lord

Different questions = Different answers ANCOVA “asks” how can one variable predict variation in another (in our example how can weight at the beginning be used to predict weight at the end) Analysis of difference scores “asks” how do initial scores differ from final scores Pike, G. R. (2004). Lord’s paradox and the assessment of change during college. Journal of College Student Development, 45 (3). 348-353.

Take Home Message When evaluating differences within individuals, there may not be a consistently best way Think of the question that you want to ask Think of what question the statistical tools can ask Be careful in interpreting your results

Thanks!