1. Alpha to Omega and Beyond! Presented by Michael Toland Educational Psychology & Dominique Zephyr Applied Statistics Lab

2. Hypothetical Experiment Suppose you measured a person’s perceived self-efficacy with the general self-efficacy scale (GSES) 1,000 times Suppose the measurements we observe vary between 13 and 17 points The person’s perceived self-efficacy score has seemingly remained constant, yet the measurements fluctuate The problem is that it is difficult to get at the true score because of random errors of measurement

3. True Score vs. Observed Score An observed score is made up of 2 components Observed Score = True score + Random Error True score is a person’s average score over repeated (necessarily hypothetical) administrations The 1,000 test scores are observed scores

4. Reliability Estimation: Classical Test Theory (CTT) Across people we define the variability among scores in a similar way S X 2 = S T 2 + S E 2 Reliability is the ratio of true-score variance relative to its observed-score variance According to CTT  xx’ = σ 2 T /σ 2 x Unfortunately, the true score variability can’t be measured directly. So, we have figured out other ways of estimating how much of a score is due to true score and measurement error

5. Concept of Reliability Reliability is not an all-or-nothing concept, but there are degrees of reliability High reliability tells us if people were retested they would probably get similar scores on different versions of a test It is a property of a set of test scores – not a test But we are not interested in just performance on a set of items

6. Why is reliability important? Reliability affects not only observed score interpretations, but poor reliability estimates can lead to deflated effect size estimates nonsignificant results When something is more reliable it is closer to the true score The more we can differentiate among individuals of different levels When something is less reliable it is further away from the true score The less it differentiates among individuals

7. Measurement Models Under CTT AssumptionsParallel Tau Equivalent Essentially Tau-EquivalentCongeneric Unidimensional YYYY Equal item covariances YYY Equal item-construct relations YYY Equal item variance YY Equal item error variance Y Alpha (  ) Cronbach (1951) Omega (  ) McDonald (1970, 1999)

8. Coefficient Alpha cov avg = average covariance among items k = the number of items ScaleV = scale score variance

9. Why have we been using alpha for so long? Simple equation Easily calculated in standard software (default) Tradition Easy to understand Researchers are not aware of other approaches

10. Problems with alpha (  ) Unrealistic to assume all items have same equal item-construct relation and item covariances are the same Underestimates population reliability coefficient when congeneric model assumed

11. What do we gain with Omega? Does not assume all items have the same item-construct relations and equal item covariances (assumptions relaxed) More consistent (precise) estimator of reliability Not as difficult to estimate as folks come to believe

12. One-Factor CFA Model Item3 33 1 Item2 22 1 Item1 11 1 Item6 66 Item5 55 Item4 44 111 1 Perceived General Self-Efficacy 1 2 3 4 5 6

13. Coefficient Omega i = factor pattern loading for item i k = the number of items  ii = unique variance of item I Assumes latent variance is fixed at 1 within CFA framework

14. Include CI along Reliability Point Estimate Measures a range that estimates the true population value for reliability, while acknowledging the uncertainty in our reliability estimate Recommended by APA and most peer reviewed journals

15. Mplus input for alpha (  )

16. Mplus output for alpha (  )

17. Mplus input for omega

18. Mplus output for omega

19. APA style write-up for coefficients with CIs  =.61, Bootstrap corrected [BC] 95% CI [.56,.66]  =.62, Bootstrap corrected [BC] 95% CI [.56,.66]

20. Limitations of CTT Reliability Coefficients and Future QIPSR Talks Although a better estimate of reliability than alpha, CTT still assumes a constant amount of reliability exists across the score continuum However, it is well known in the measurement community that reliability/precision is conditional on a person’s location along the continuum Modern measurement techniques such as Item Response Theory (IRT) do not make this assumption and focus on items instead of the total scale itself

21. References Revelle, W., Zinbarg, R. E. (2008). Coefficients Alpha, Beta, Omega, and the glb: Comments on Sijtsma. Psychometrika, 74, 145-154. doi:10.1007/s11336-008-9102-z Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research and Evaluation, 17. Retrieved from: http://pareonline.net/getvn.asp?v=17&n=3http://pareonline.net/getvn.asp?v=17&n=3 Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6. Retrieved from: http://digitalcommons.wayne.edu/jmasm/vol6/iss1/4 http://digitalcommons.wayne.edu/jmasm/vol6/iss1/4 Starkweather, J. (2012). Step out of the past: Stop using coefficient alpha; there are better ways to calculate reliability. Benchmarks RSS Matters Retrieved from: http://web3.unt.edu/benchmarks/issues/2012/06/rss-mattershttp://web3.unt.edu/benchmarks/issues/2012/06/rss-matters Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74, 107-120.doi:10.1007/s11336-008-9101-0 Peters., G-J. Y. (2014). The alpha and the omega of scale reliability and validity: Why and how to abandon Cronbach's alpha and the route towards more comprehensive assessment of scale quality. The European Health Psychologist, 16, 56–69. Retrieved from: http://www.ehps.net/ehp/issues/2014/v16iss2April2014/4%20%20Peters%2016_2_EHP_April%202014.pdf http://www.ehps.net/ehp/issues/2014/v16iss2April2014/4%20%20Peters%2016_2_EHP_April%202014.pdf Crutzen, R. (2014). Time is a jailer: What do alpha and its alternatives tell us about reliability?. The European Health Psychologist, 16, 70-74. Retrieved from: http://www.ehps.net/ehp/issues/2014/v16iss2April2014/5%20Crutzen%2016_2_EHP_April%202014.pdf http://www.ehps.net/ehp/issues/2014/v16iss2April2014/5%20Crutzen%2016_2_EHP_April%202014.pdf Dunn, T., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105, 399-412. doi:10.1111/bjop.12046 Geldhof, G., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19, 72-91. doi:10.1037/a0032138

22. Acknowledgements APS Lab members Angela Tobmbari Zijia Li Caihong Li Abbey Love Mikah Pritchard