1. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. 2/3/2016 Detection of Unusual Administrations Using a Linear Mixed Effects Model Minzhao Liu, University of Florida Mentors: Yi-Hsuan Lee Alina von Davier

2. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. Outline 2/3/2016 Design of Experiment Model and Method Results and Interpretation Prediction Discussion 2

3. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. Objectives Detect unusual administrations based on the model we fit Use regression model to fit the data Hypothesis: individual scores can be explained by background predictors 2/3/2016 3

4. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. Illustration of the Hypothesis 2/3/2016 4 Gender Country Repeater Years of study Daily study time Level of education Major Yrs abroad (Eng) Status Individual score Hypothetical Distribution of individual score Mean score Of subgroup Distribution of subgroup mean score

5. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. Data Description 2/3/2016 TOEIC data 15 administrations Random sample of 18,000 test- takers per administration 16 background questions 5

6. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. 2/3/2016 6 Design of Experiment TOEIC data 15 admin 18,000 test-takers per admin 16 background variables 3 groups Country A Country B Equated Univariate responses: Reading scores Listening scores An equating plan

7. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. Linear Mixed Effects Model 2/3/2016 Normality assumption (checked through residual diagnostics) Random effects can explain correlation among individuals In general: 7

8. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. Why Mixed Model 2/3/2016 8 Group random effect Admin random effect Individual error Fixed effects (background variables)

9. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. Why Mixed Model (cont’d) 2/3/2016 9

10. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. Variable Selection Forward selection 3 steps: 1.Random effects 2.Main effects 3.Interactions criteria 2/3/2016 10

11. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. Final Models Reading: 1.Years of study 2.Yrs abroad (Eng) 3.Repeater/First time 4.Level of Education 5.Major Random: admin 2/3/2016 11 Listening: 1.Years of study 2.Yrs abroad (Eng) 3.Repeater/First time 4.Level of Education 5.Major 6.Daily study time Random: admin Note: ‘group’ and ‘country’ variables did NOT turn out to be significant.

12. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. Prediction Definition of ‘Unusual’ Distribution of mean score of certain admin Approximate 95% prediction interval Example: for administration 1, the observed listening mean score is 317.3, and the prediction interval is (301.7, 350.0) 2/3/2016 12

13. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. Discussion Good news: 1.No significant effect for ‘country’ 2.No significant effect for ‘group’ 3.Prediction interval is narrow Limitations: – Nonlinearity – Few groups and administrations – Seasonality – Country and admin are confounded in the design 2/3/2016 13

14. Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved. 2/3/2016 14 Thank You for Your Attention~ Go Gators !