1. Data Forensics: A Compare and Contrast Analysis of Multiple Methods Christie Plackner

2. Outlier Score Applied to most of the methods Statistical probabilities were transformed into a score of 0 to 50 10 = statistically unusual 2

3. Erasure Analysis Wrong-to-right (WR) erasure rate higher than expected from random events The baseline for the erasure analysis is the state average One sample t-test 3

4. Scale Score Changes Scale score changes statistically higher or lower than the previous year Cohort and Non-cohort One sample t-test 4

5. Performance Level Changes Large changes in proportion in performance levels across years Cohort and Non-cohort Log odds ratio –adjusted to accommodate small sample size –z test 5

6. Measurement Model Misfit Performed better or worse than expected Rasch residuals summed across operational items Adjusted for unequal school sizes 6

7. Subject Regression Large deviations from expected scores Within year – reading and mathematics Across year – cohort within a subject One sample t-test 7

8. Modified Jacob and Levitt Only method not resulting in a school receiving a score Combination of two indicators: –unexpected test score fluctuations across years using a cohort of students, and –unexpected patterns in student answers Modified application of Jacob and Levitt (2003) –2 years of data –Sample size 8

9. Principal Component Analysis Does each method contribute to the overall explained variance? Can the methods be reduced for a more efficient approach? 9

10. Multiple Methods 1.Erasure Analysis (mER) 2.Scale score changes using non-cohort groups (mSS) 3.Scale score changes using cohort groups (mSC) 4.Performance level changes using non-cohort groups (mPL) 5.Performance level changes using cohort groups (mPLC) 6.Model misfit using Rasch Residuals (mRR) 7.Across subject regression using reading scores to predict mathematic scores (mRG) 8.Within subject regression using a cohort’s previous year score to predict current score (mCR) 9.Index 1 of the Modified Jacob and Levitt evaluating score changes (mMJL1) 10.Index 2 of the Modified Jacob and Levitt evaluating answer sheet patterns (mMJL2). 10

11. Principal Component Analysis Grade 4 mathematics exam 10 methods Method MeanStd. DeviationAnalysis N mSS2.492.37711692 mPL1.751.48361692 mRG1.352.51611692 mRR1.01.85061692 mER1.5803.73751692 mSC3.4293.91411692 mPLC1.4861.49271692 mMJL1.50278.2883191692 mMJL2.49774.2877031692 mCR4.44955.146031692 11

12. Method Correlations mSSmPLmRGmRRmERmSCmPLCmMJL1mMJL2mCR mSS1.000 mPL.5031.000 mRG.029-.0421.000 mRR-.033-.020-.0331.000 mER.084.034.070.1281.000 mSC.121.048.322.050.0871.000 mPLC.085.184-.010.056.094.4651.000 mMJL1-.028-.078.518-.054.092-.045-.1221.000 mMJL2.097.006.133.235.212.400.217-.0051.000 mCR.135.058.285.056.091.986.474-.107.4051.000 12

13. Principal Component Statistics Component Initial Eigenvalues Extraction Sums of Squared Loadings Total% of VarianceCumulative %Total% of VarianceCumulative % 12.75827.582 2.75827.582 21.62516.24743.8291.62516.24743.829 31.40714.07457.9021.40714.07457.902 41.20412.04569.9471.20412.04569.947 5.8478.47178.418 6.7247.23885.656 7.6046.04091.696 8.4594.58796.283 9.3603.60199.884 10.012.116100.000 13

14. Scree Plot 14

15. Loading Matrix Component 12345678910 mCR.924.026-.147-.230.013-.086-.193-.017.140.077 mSC.923.079-.125-.236.019-.069-.183-.015.162-.076 mPLC.617-.266-.126-.120-.206.576.291.176-.168.000 mMJL2.592.065-.147.405.117-.390.535-.083-.040.000 mMJL1-.023.732.476.085.031.234.205.142.334.005 mRG.359.706.385-.089.147.031-.123-.143-.397.000 mSS.267-.438.679.095.122-.236-.086.428-.074.000 mPL.194-.555.636.046.108.180.050-.438.0977.526E-5 mRR.158-.033-.282.688.533.288-.227.048.013.000 mER.239.088.115.646-.676-.016-.209-.048-.007.000 15

16. Simplified Loading Matrix +/- greater than 1/2 the maximum value in the component (+)/(-) is between ¼ to ½ the maximum 16 1234 mCR+(-) mSC+(-) mPLC+(-) mMJL2++ mMJL1+.+ mRG(+)++ mSS(+)_+ mPL_+ mRR(-)+ mER(+)+

17. Principal Component Statistics Component Initial Eigenvalues Extraction Sums of Squared Loadings Total% of VarianceCumulative %Total% of VarianceCumulative % 12.75827.582 2.75827.582 21.62516.24743.8291.62516.24743.829 31.40714.07457.9021.40714.07457.902 41.20412.04569.9471.20412.04569.947 5.8478.47178.418 6.7247.23885.656 7.6046.04091.696 8.4594.58796.283 9.3603.60199.884 10.012.116100.000 17

18. Scree Plot 18

19. Reducing Variable Set Determine how many components to retain –Cumulative percentage of total variation –Eigenvalues –The scree plot Method Number of Retained Components 90% Cumulative Variance7 70% Cumulative Variance4 Eigenvalue4 Scree Plot2 19

20. Reducing Variable Set Select one method to represent a component Selecting methods within components –Positive selection Retain highest loading method with components –Discarded principal components Remove highest loading method with 20

21. Reducing Variable Set Selection Method Positive Selection Discarded Principal Components Number of Components 4242 mCRXXXX mSC mPLC mMJL2 mMJL1XXXX mRG mSSXX mPL mRRXX mER Cohort regression* Modified J&L, Index 1* Non-cohort scale score change Model misfit 21

22. Conclusion All methods seem to account for variation in detecting test taking irregularities Accounting for the most –Cohort regression –Cohort scale score change –Cohort performance level change Method reduction results the same 22

23. Discussion Different component selection methodologies Closer examination of variables –Remove cohort regression or cohort scale score change –Combine the J&L indexes Remove erasures 23