1. ® The Use of Confirmatory Factor Analysis to Support the Use of Subscores for a Large-Scale Student Outcomes Assessment Rochelle S. Michel, Ph.D. Evaluation 2009 Annual Conference of the American Evaluation Association November 11-14, 2009

2. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 2 Introduction Institutions consistently request assessment information beyond the total score –Subscores –Item analysis –Diagnostic information –Comparative data

3. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 3 Introduction Standard A4 – Program Evaluation Standards (1994) –Calls for the use of valid information in the evaluation of programs, establishing the validity of inferences drawn from information obtained in relation to the evaluation questions (p.147)

4. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 4 Subscores “Easiest” information that can be provided Often used as the data by which to measure and document program effectiveness Research on issues related to subscore reporting –Haberman, 2005; Haberman, Sinharay, Puhan, 2007; High correlation between subscores and total score Additional information of subscores beyond total score

5. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 5 General Education Measure 108 multiple-choice items 4 skills (each measured by 27 items) –Reading –Written communication –Critical thinking –Mathematics 3 contexts (each measured by 18 items) –Humanities –Social Science –Natural Science

6. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 6 Research Questions How well does a four-factor model fit the data from the general education student outcomes measure? How well does a one-factor model fit the data from the general education student outcomes measure? What do the results of the CFA indicate about subscore usage for the measure under analysis?

7. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 7 Demographics Test takers (n=15,504) –Gender Females – 59% Males – 41% –Ethnicity White – 75% African American – 12% Hispanic – 4% Institutions (n=54)

8. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 8 Psychometric Properties Test reliability, 0.94 Skills –Reading,.85 –Critical Thinking,.80 –Writing,.77 –Math,.85 Contexts –Natural Sciences,.80 –Social Sciences,.77 –Humanities,.71

9. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 9 Correlations among skills SkillsTotalRDCTWM Total1.00 RD.921.00 CT.90.831.00 W.86.75.691.00 M.84.66.64.621.00 Note. RD=Reading, CT=Critical Thinking, W=Writing, M=Mathematics

10. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 10 Correlations among contexts ContextsTotalHumanitiesNatural Sciences Social Sciences Total1.00 Humanities.841.00 Natural Sciences.89.741.00 Social Sciences.87.73.781.00

11. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 11 Confirmatory Factor Analysis Analyses performed using LISREL 8.8 (Joreskog & Sorbom, 2006) Used the tetrachoric correlation matrix to create the covariance matrix to be analyzed Chi-square statistic –Traditional measure for evaluating overall model fit –Hypothesizes that the model fits the population data perfectly –Both one-factor and four-factor models had statistically significant chi-square values

12. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 12 Confirmatory Factor Analysis Summary of Fit Indices for the One-Factor and Four-Factor Models of a General Education Measure of Student Outcomes SampleRMSEAECVISRMRGFICFINNFI M10.052816.1980.03860.7700.9760.975 M40.03898.9640.03230.8580.983 Note. M1=one-factor model; M4=four-factor model; RMSEA=root mean square error of approximation; ECVI=expected cross-validation index; SRMR=standardized root mean square residual; GFI=goodness-of-fit index; CFI=comparative fit index; NNFI=non-normed fit index.

13. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 13 Summary Results are consistent with previous exploratory factor analysis conducted –Conflicting results between various fit indices –Neither one-factor nor four-factor model provide an appropriate fit to the data

14. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 14 Implications Examination of alternative approaches to subscore reporting –Use of scale anchoring methodology/proficiency scaling Responsible use of current subscores –Total score may provide more information about test takers’ skills than the subscores –Care should be taken making decisions based solely on skill subscores

15. ® Confidential and Proprietary. Copyright © 2009 Educational Testing Service. All rights reserved. 15 Thank you! Thank you to Michael Chajewski and Veleka Allen who performed some of the preliminary analyses on the measure. Contact: rmichel@ets.org