1. Principles of Inter-rater Reliability (IRR) Dr. Daniel R. Winder

2. NEW IRR for each NEW study Inter-rater reliability of judges scores must be demonstrated for each additional study even if the study is using an instrument that has been validated in previous studies. WHY?

3. NEW IRR for each NEW study Because the correlation coefficients were established on previous judges scores, not the current judges. Principle: IRR of judge’s scores is sample dependent. Therefore, it is considered good practice to report IRR in all studies involving more than one rater.

4. Why IRR? 1) To show agreement of judges scores. Why? Personnel ratings are expensive—if the judges agree on the ratings, only one will have to rate, saving time, money, resources, etc. However, a good practice is to have judges overlap on their ratings (e.g. with 24 cases, one rates 1-8, the other rates 17-24, both rate 9-16).

5. Why IRR? 2) To show the validity of scores from a method, a rubric, judge, or other instrument. Why? If the rubric is consistent for a sample of several different raters, then it may be generalizable to the population the raters come from. This is more of a generalizability question for which a G-study or D-study will be utilized (more later).

6. Why IRR? 3) To identify problem areas of a rubric or scoring method. Where is the inconsistency coming from? (untrained raters, occasions, unexplained error, outliers, etc).

7. Three types of IRR 1.Agreement/consensus of scores 2.Consistency of scores 3.Measurement of scores Notice that IRR is about scores, not the construct (If we were focusing on the construct, we would be focusing on the validity, not reliability).

8. Agreement of Raters (i.e. a classification consensus) When data is nominal, use an agreement coefficient. For example, when classifying a type of behavior (schizophrenia) rather than a quantity of a behavior (dexterity of left hand). Example: Do the doctors agree that the type of irrational mental behavior is the same rather than the degree of irrational mental behavior? Example: Grading themes in writing vs. grading quality of writing skills. Example: Does this business qualify as an S- corp, LLC, non-profit organization, etc vs. is it profitable. Agreement coefficients can be use to measure reliability of ordered likert-data (e.g. 1-5 ordered ratings), but there are more powerful methods for analyzing this type of ordinal-level data than mere agreement.

9. How to compute agreement Number of agreement of cases divided by total number of cases. Usually reported as percentages. –E.g. Rater 1 and 2’s scores agree on 60 of the 80 cases. –60/80 =.75 –Thus, 75% agreement.

10. Adjacent Agreement Used when looking at Likert-scale agreement with a large number of categories. You relax a bit and give equivalence to scores in adjacent categories (if I gave a 4 and the other judge gave a 5, we generally agree, so we count this as agreement, dude). Why should this not be used on scales of 5 or less categories? Because almost all categories are adjacent.

11. Agreement Guidelines Scores are proportions between 0 and 1, usually made percentages. 0 = no agreement, 1 = perfect agreement Quality of inter-rater reliability should be 70% or greater (Stemler, Steven E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9(4). Retrieved March 22, 2010 from http://PAREonline.net/getvn.asp?v=9&n=4.)

12. Agreement-Cohen’s Kappa (most common IRR stat) The big hoorah of this stat is that it takes into account the amount of agreement that would occur from chance alone. For example, with 5 equally used categories, there is a 20% chance that agreement would occur by chance alone. So is it more than chance, that my scores agree with another raters? (Cohen, 1960, 1968).

13. Agreement-Cohen’s Kappa (most common IRR stat) If concerned that agreement is due to lack of options in rating categories this is helpful. Generally, less than 5 categories.

14. Agreement-Kendall’s W Use this when data is non-parametric (not normally distributed). Suppose, for instance, that a number of people have been asked to rank a list of political concerns, from most important to least important. Kendall's W can be calculated from these data. If the test statistic W is 1, then all the survey respondents have been unanimous, and each respondent has assigned the same order to the list of concerns. If W is 0, then there is no overall trend of agreement among the respondents, and their responses may be regarded as essentially random. Intermediate values of W indicate a greater or lesser degree of unanimity among the various responses. While tests using the standard Pearson correlation coefficient assume normally distributed values and compare two sequences of outcomes at a time, Kendall's W makes no assumptions regarding the nature of the probability distribution and can handle any number of distinct outcomes. http://en.wikipedia.org/wiki/Kendall's_W

15. Agreement-Guidelines 0 = agreement is no greater than what would occur by chance alone. What would happen if judges agreed less than chance alone would predict? (a negative value). Moderate values = 0.41–0.60, Substantial values = 0.60 or > (Landis and Koch,1977) Report IRR agreement stats between all judges or for several judges report the minimum, mean, and maximum IRR stat.

16. Problems with Agreement What if one of the judges exhibits a leniency factor (e.g. he/she is consistently 2 points higher?). What would this systematic difference do to an agreement statistic? (it would misrepresent the ratings by reporting a low IRR when in fact, the scores covary quite nicely and one score can predict the other score quite well—we overcome this by using consistency rather than agreement/consensus stats).

17. Consistency Stats Most useful when using continuous data and reported raw scores are not used as cut-off scores. However, if the instrument from a raw summed score, say a 70 out of 80, is a cut-off score for diagnosing a disability or other diagnosis, then you should be concerned more with agreement than consistency (i.e. go back to the previous slides on agreement and follow those guidelines/methods). Consistency is important when the main thing that a scale measures is rank-order. That said, many report agreement and consistency.

18. Consistency Stats Useful for scales with categories that measure unidimensional traits (one thing) and higher categories represent more of the trait. Useful when training judges is not practical or not important (because judges can develop their own interpretation of the severity or leniency of the rating scale as long as they are consistent with themselves—intra-rater reliability is more important than agreement). Some methods can handle multiple raters rather than only two at a time (cronbach’s alpha).

19. Consistency Stats Could two scores agree but not be consistent? (generally, no) Could two scores be consistent but not agree? (yes). Could two raters means and/or medians be significantly different but have high consistency? (yes—see Stemler 2004 for the rare exception) Consistency stats also identify whether the scores can be corrected by a lenient of severe judge For example, if one judge consistently rates 2 points lower than all other judges and the consistency stat shows a high consistency, then it is worth it to look into covariance to decide if you should correct the severity factor in this judge (i.e. if one judge’s scores are always 2 points lower, you may want to bump those scores up to compare with other judges for agreement).

20. Consistency Stat-Pearson Correlation Coefficient Assumes data is normally distributed. Because data is continuous, scores can be in between categories (1.5 vs. 1 or 2) Can easily be computed by hand (Glass & Hopkins, 1996). Limited by only being able to be computed for one pair of judges at a time and only one item at a time.

21. Consistency Stat Intra Class Coefficient (ICC) Quite common among OT. Similar to Pearson’s r but allows multiple raters.

22. Consistency Stat Spearman Rank Coefficient Can be used for non-normally distributed data. Ratings can be correlated based on rank-order agreement (Crocker & Algina, 1986). However, all raters have to rate all cases, can only compare two ordinal rated sets at a time. Kendall's tau: Another common correlation for use with two ordinal variables or an ordinal and an interval variable. Prior to computers, rho was preferred to tau due to computational ease. Now that computers have rendered calculation trivial, tau is generally preferred.

23. Consistency Stat Cronbach’s Alpha Measures observed rating + error divided by estimated true score. Thus a proportion between 0 and 1 ensues (Crocker & Algina, 1986). What would happen if there was little error? (close to 1, much agreement due to little error) What would happen if there was a lot of error? (close to 0, no agreement due to error) Used when multiple judges rate so all scores can be analyzed together to check if observed ratings are due to consistency or error. Limited-every judge must rate every case.

24. Do not use consistency stat when… Nominal data It’s NOT okay for judges to agree to disagree in the rating number as long as they are consistent with themselves (e.g. the total raw score is used as a cut-off for a diagnosis or conclusion). judges have severe differences in variability of categories used (one uses many 1-7 categories and another only uses 1-3). The lack of variance cannot be corrected by a mean adjustment. Most of the ratings fall into one or two categories. In this case, the correlation coefficient may be deflated due to lack of variability rather than lack of agreement.

25. Consistency Stat Guidelines Values greater than.70 are acceptable (Barrat, 2001).

26. Measurement Stats Assumes that information from all the judges, whether they agree or not, is valuable information. Linacre suggests that training can bias results but measurement stats don’t require so much training that it will bias results (2004). Use when different levels of the trait are represented in different levels of the unidimensional scale (for example a 1 means that a person was rated lower than 3). Use when multiple judges rate different but not all cases or items.

27. Measurement Stats Factor Analysis Determines the amount of shared variance from rater’s scores that is attributed to a single factor. If the amount of shared variance from that factor is high (it explains 60% of the rater’s scores variance), this adds evidence that the raters are consistently rating the same construct. Once this factor has been established, each case can receive a score based solely on how well their score predict this single factor. Harman, 1967

28. Measurement Stats Generalizability Theory The goal of generalizability theory is to separate the variance components of raters scores. These components can be based on persons, raters, occasions, or other components of a rating as well as unexplained error. When you know where the variance is coming from, you can isolate these components to get a more stable comparison of agreement without the confounding component. Predictive formulas can be used to determine how many raters, occasions, and compenents should be considered to reduce error to an optimal low rate. Shavelson and Webb, 1991

29. Generalizability Theory—parsing out variance components Person Occasion Rater Rater x Occasion interaction Person x Rater interaction Person x Occasion interaction Error or unexplained variance Shavelson and Webb, 1991

30. D study Using generalizability theory, we can determine how many components should be studied as well as how many raters, occasions, etc are needed to get an optimal rating (cost benefit analysis—if I add three more judges, how much error do I eliminate—is it worth it.).

31. www.courseoutcomes.com Visit www.courseoutcomes.com to download a free tool for Generalizability studies. This tool conducts a G-study and D-study for you and teaches the basic concepts of Generalizability Theory.www.courseoutcomes.com

32. Measurement Stats Many-facets Rasch Model This model allows for multiple raters who don’t have to rate the same items or persons. Essentially, uses a logit scale to determine the difficulty of getting different scores from different judges. For example, how much harder is it to get an “18” from rater 1 vs. rater 5? Is an 18 from rater 5 like getting an 15 from rater 1? Let’s you know which items are the most difficult and the least difficult. Should be used when it is assumed that a ratee’s ability may play into the accuracy of the score. Yields reliability statistics for the overall trait/scale. Linacre, 1994; Rasch, 1960/1980; Wright & Stone, 1979; Bond and Fox, 2001

33. Measurement Stats Many-facets Rasch Model Fit statistics tell you whether the a judge is consistent with their own rating scale across persons and items. Measurement errors are taken at each level of the trait rather than assuming that they remain constant across the whole scale or trait. For example, a scale may have less error for persons in the middle of the scale then those on the ends, thus it would be more reliable to rate persons who are in the middle of the ability or trait rather than the ends of the trait. Linacre, 1994; Rasch, 1960/1980; Wright & Stone, 1979; Bond and Fox, 2001

34. Measurement Stats Problems You need specialized knowledge to run programs as these cannot be computed by hand. In addition, you need specialized knowledge to interpret the outputs of programs and conceptually make sense of the output. Only works with ordinal data. Linacre, 1994; Rasch, 1960/1980; Wright & Stone, 1979; Bond and Fox, 2001

35. The most clear IRR article Stemler, Steven E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9(4). Retrieved March 22, 2010 from http://PAREonline.net/getvn.asp?v=9&n=4. Much of this training follows Dr. Stemler’s format.

36. Bibliography (I have only referenced in the slides the ones I directly used but I have left all of these as resources). Barrett, P. (2001, March). Assessing the reliability of rating data. Retrieved June 16, 2003, from http://www.liv.ac.uk/~pbarrett/rater.pdfhttp://www.liv.ac.uk/~pbarrett/rater.pdf Bock, R., Brennan, R. L., & Muraki, E. (2002). The information in multiple ratings. Applied Psychological Measurement, 26(4), 364-375. Bond, T., & Fox, C. (2001). Applying the Rasch model. Mahwaw, NJ: Lawrence Erlbaum Associates. Burke, M. J., & Dunlap, W. P. (2002). Estimating interrater agreement with the average deviation index: A user's guide. Organizational Research Methods, 5(2), 159-172. Cohen, J. (1960). A coefficient for agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46. Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scale disagreement or partial credit. Psychological Bulletin, 70, 213-220. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (Third ed.). Mahwah, NJ: Lawrence Erlbaum Associates. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Harcourt Brace Jovanovich. Glass, G. V., & Hopkins, K. H. (1996). Statistical methods in education and psychology. Boston: Allyn and Bacon. Harman, H. H. (1967). Modern factor analysis. Chicago: University of Chicago Press. Hayes, J. R., & Hatch, J. A. (1999). Issues in measuring reliability: Correlation versus percentage of agreement. Written Communication, 16(3), 354-367. Hopkins, K. H. (1998). Educational and psychological measurement and evaluation (Eighth ed.). Boston: Allyn and Bacon. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174. LeBreton, J. M., Burgess, J. R., Kaiser, R. B., Atchley, E., & James, L. R. (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar? Organizational Research Methods, 6(1), 80-128. Linacre, J. M. (1988). FACETS: a computer program for many-facet Rasch measurement (Version 3.3.0). Chicago: MESA Press. Linacre, J. M. (1994). Many-facet Rasch measurement. Chicago: MESA Press. Linacre, J. M. (2002). Judge ratings with forced agreement. Rasch Measurement Transactions, 16(1), 857-858. Linacre, J. M., Englehard, G., Tatem, D. S., & Myford, C. M. (1994). Measurement with judges: many-faceted conjoint measurement. International Journal of Educational Research, 21(4), 569-577. Mertler, C. A. (2001). Designing scoring rubrics for your classroom. Practical Assessment, Research and Evaluation, 7(25). Moskal, B. M., & Leydens, J. A. (2000). Scoring rubric development: Validity and reliability. Practical Assessment, Research and Evaluation, 7(10). Myford, C. M., & Cline, F. (2002, April 1-5). Looking for patterns in disagreements: A Facets analysis of human raters' and e-raters' scores on essays written for the Graduate Management Admission Test (GMAT). Paper presented at the Annual meeting of the American Educational Research Association, New Orleans, LA. Rasch, G. (1960/1980). Probabilistic models for some intelligence and attainment tests (Expanded ed.). Chicago: University of Chicago Press. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage Publications. Stemler, S. E. (2001). An overview of content analysis. Practical Assessment, Research and Evaluation, 7(17), Available online: http://PAREonline.net/getvn.asp?v=7&n=17. http://PAREonline.net/getvn.asp?v=7&n=17 Stemler, S. E., & Bebell, D. (1999, April). An empirical approach to understanding and analyzing the mission statements of selected educational institutions. Paper presented at the New England Educational Research Organization (NEERO), Portsmouth, NH. Tierney, R., & Simon, M. (2004). What's still wrong with rubrics: Focusing on consistency of performance criteria across scale levels. Practical Assessment, Research & Evaluation, 9(2), Retreived February 16, 2004 from http://PAREonline.net/getvn.asp?v=9&n=2.http://PAREonline.net/getvn.asp?v=9&n=2 Uebersax, J. (1987). Diversity of decision-making models and the measurement of interrater agreement. Psychological Bulletin, 101(1), 140-146. Uebersax, J. (2002). Statistical methods for rater agreement. Retrieved August 9, 2002, from http://ourworld.compuserve.com/homepages/jsuebersax/agree.htmhttp://ourworld.compuserve.com/homepages/jsuebersax/agree.htm Winer, B. J. (1962). Statistical principals in experimental design. New York: McGraw-Hill. Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago: MESA.