Coding with R-PAS: Does Prior Training with the Exner Comprehensive System Impact Interrater Reliability Compared to Those Examiners with Only R-PAS Based Training? Jennifer H. Lewey, M.A., Trisha M. Kivisalu, M.A., Thomas W. Shaffer, Ph.D., ABPP, & Merle L. Canfield, Ph.D. California School of Professional Psychology at Alliant International University Introduction The Rorschach Inkblot Method (RIM) is a free response personality assessment instrument widely used in both clinical and research settings worldwide. While multiple coding systems exist, the predominant system utilized has been the Comprehensive System (CS). Published in 2011 the Rorschach-Performance Assessment System (R-PAS) provides detailed guidelines for test administration and coding responses. There are a number of variable differences between these coding systems. Response-level agreement is essential for determining precision related to coders’ ability to apply the same categorical coding scheme to responses, and ascertain useful information for monitoring training and practice. Interrater agreement research is important in that it reflects the degree of consistency among raters, which serves as a precursor for accuracy in using the Rorschach as a clinical assessment tool (Sahly, Shaffer, Erdberg, & O’Toole, 2011). This investigation examines interrater reliability between two unique examiner groups: (1) those graduate student examiners with previous Comprehensive System training background who are now trained in R-PAS, and (2) those graduate students trained solely with the R-PAS. The purpose of this study was two-fold: (1) to ascertain if there was a difference in interrater reliability between these with varied training, and (2) to identify particular strengths and weaknesses that could lead to informing future instruction on coding with these two interpretive systems for the Rorschach in clinical and academic settings. Methodology The current study analyzed a non-clinical convenience sample of 16 participants representing a total of 364 responses. All examiners were psychology doctoral students enrolled in an advanced research practicum course at Alliant International University in Fresno, California. Data for the sample was collected from January 2013 through May 2014, as part of a normative data collection study. Graduate students utilized a blind coding method and any differences were resolved in order to achieve interrater agreement. Discrepancies between the initial and blind coders’ ratings were analyzed for each variable with SPSS and expressed as percent agreement and kappa values. Table 2 Cicchetti’s Guidelines for Interpreting Kappa (κ) Results Findings for each value are presented in Table 1. Cicchetti’s (1994) guidelines, presented in Table 2, were used for interpreting kappa values. Overall, rates of agreement and kappa values were mixed for both examiners who were trained in CS & R-PAS and solely R-PAS (See Table 3). Table 3 Select Variables Highlighting the Respective Training with Higher Concordance Rates *Note: *Denotes summation of variables for the Location category. Discussion In summary, the findings indicate minimal interrater reliability differences across those trained in CS and R-PAS and solely R-PAS suggesting that regardless of prior training, when using the detailed coding system designed by R-PAS developers, R-PAS variables can be coded reliably among raters. In particular, select variables may have higher concordance rates due to familiarization with the same variables in the Comprehensive System. Conversely, this may confound reliability rates for other variables that exist solely in R-PAS. These results emphasize the importance of training and blind coder practice to ensure coding accuracy for this empirically based system of coding. Overall, interrater reliability is essential to ensure accuracy of coding and provides insights for future training with graduate students and professionals. Kappa Values CategoryPoorFairGoodExcellent Value< – 1.00 Table 1 Comparison of CS & R-PAS versus R-PAS Trained Only Rorschach Interrater Agreement CS & R-PASR-PAS Only VariableAgreement (%)κ κ Location W D Dd Dd Space SR SI Content Class H (H) Hd (Hd) A (A) Ad (Ad) An Art Ay Bl Cg Ex Fi Sx NC Synthesis/Vagueness Sy Vg Pair Form Quality FQo FQu FQ Popular P Determinants M FM m a p FC CF C C’ V Y T--- FD r F Cognitive Codes DV DR DR INC INC FAB FAB PEC Thematic Codes ABS PER COP MAH MAP AGM AGC MOR ODL VariableCS & R-PAS R-PAS Only Location Location* SpaceSISR Content Class(H)Hd (A)(Hd) AyArt SxCg NC Synthesis/VaguenessVgSy Form QualityFQu PopularP Determinants A FC CF MC RC’ Thematic CodesAGMMAH ODLAGC