Download presentation

Presentation is loading. Please wait.

Published byEmilio Hueston Modified over 2 years ago

1
Exploring new versions of DIMTEST for use with Polytomous Data Tan Li Louis Roussos Measured Progress July 24, 2009

2
Outline Introduction Dimensionality Hypothesis Test Poly-DIMTEST Methods Poly-DIMTEST without AT2 Poly-NEWDIM Simulation Study Results & Conclusions Future Work

3
Introduction Dimensionality Hypothesis Test Important assumption for many IRT models Equating Scoring Scaling Calibration DIF analysis Hypothesis Test H 0 : d E = 1 vs. H 1 : d E > 1

4
Introduction Poly-DIMTEST ( Nandakumar, Yu, Li, & Stout, 1998 ) Hypothesis test H 0 : vs. H 1 : Split all the test items into three subtests: AT1, AT2 and PT The test statistic: Stand Error of CCOV comes from a complicated formula

5
Introduction Poly-DIMTEST Weaknesses Difficulty on finding and choosing AT2 items Not enough items left for PT

6
Methods Poly-DIMTEST without AT2 Based on dichotomous version of DIMTEST without AT2 (Stout, Froelich, & Gao, 2001) Steps AT and PT 1. Split all the test items into two subtests: AT and PT 2. Fit a unidimensional nonparamatric model to the original data by kernel smoothing take the place ofAT2 3. Simulate N samples from the model to take the place of AT2 The test statistic: Stand Error of CCOV comes from the same formula provided by Nandakumar, et al. (1998)

7
Methods Poly-NEWDIM Based on dichotomous version of NEWDIM (Seo & Roussos, 2009) Similar procedure with Poly-DIMTEST without AT2 The test statistic: Standard Error Standard Error comes from the Standard Deviation over the simulated samples

8
Simulation Study Dichotomous items All of the parameters were randomly generated from the distributions based on real data from a large multi-year pool of 729 grade 5 math items

9
Simulation Study Polytomous items All of the parameters were randomly generated from the distributions based on real data from a large multi-year pool of 729 grade 5 math items

10
Simulation Study Type I Error Study Power Study 2 dimensions simple structure

11
Simulation Study Factors 500 examinees and 1000 examinees 52 pts test and 32 pts test AT subtest 52pts test: 5 MC, 10 MC, 2 CR, and 5 CR items 32pts test: 3 MC, 6 MC, and 3 CR items 52 pts 32 pts Dich Poly

12
Results Type I Error for 52 points test Sample sizePD-NO-AT2PND Average Type I Error for 32 points test Sample sizePD-NO-AT2PND Average Power for 52 points test Sample SizePD-NO-AT2PND Average Power for 32 points test Sample SizePD-NO-AT2PND Average

13
Results Type I Error – 52 pts test, 400 trials Poly-DIMTEST without AT2 D,P5 MC10 MC2 CR5CR 52, , , , Poly-NEWDIM D,P 5 MC10 MC2 CR5CR 52, , , , Examinees 7 Poly-DIMTEST without AT2 D,P5 MC10 MC2 CR5CR 52, , , ,13 01 Poly-NEWDIM D,P 5 MC10 MC2 CR5CR 52, , , , Examinees

14
Results Type I Error – 32 pts test, 400 trials Poly-DIMTEST without AT2 D,P 3 MC6 MC3 CR 32,012 20, , , Poly-NEWDIM D,P3 MC6 MC3 CR 32, , , , Examinees 7 Poly-DIMTEST without AT2 D,P 3 MC6 MC3 CR 32, , ,6001 0,8 0 Poly-NEWDIM D,P 3 MC6 MC3 CR 32,066 20, , , Examinees

15
Results Power – 52 pts test,400 trials Poly-DIMTEST without AT2 D,P 5 MC10 MC2 CR5CR 52, , , , Poly-NEWDIM D,P5 MC10 MC2 CR5CR 52, , , , Examinees < 85 ≥ Examinees Poly-DIMTEST without AT2 D,P5 MC10 MC2 CR5CR 52, , , , Poly-NEWDIM D,P5 MC10 MC2 CR5CR 52, , , ,

16
Results Power – 32 pts test, 400 trials Poly-DIMTEST without AT2 D,P 3 MC6 MC3 CR 32, , , ,8 89 Poly-NEWDIM D,P 3 MC6 MC3 CR 32, , , , Examinees < 85 ≥85 Poly-DIMTEST without AT2 D,P3 MC6 MC3 CR 32, , , , Poly-NEWDIM D,P3 MC6 MC3 CR 32, , , , Examinees

17
Conclusion Type I error study Conservative Type I error behavior Poly-NEWDIM performs closer to nominal (0.05). Power study Poly-NEWDIM has greater power than Poly- DIMTEST without AT2 Poly-NEWDIM provides adequate power for a variety of conditions.

18
Future Work More examinees Dimensionality structure Item parameter simulation models Develop a method to choose AT subtest for mixed MC and CR tests Real datasets Skewed ability distributions

19
Reference Nandakumar, R., Yu, F., Li, H., & Stout, W. (1998). Assessing Unidimensionality of Polytomous Data. Applied Psychological Measurement, 22, Stout, W., Froelich, A., & Gao, F. (2001). Using Resampling Methods to Produce an Improved DIMTEST Procedure. Essays on item response theory, Seo, M., & Roussos, L. (2009). Evaluation of DIMTEST Effect-Size Measure and Its Application. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego.

20
Acknowledgement Measured Progress Department of Psychometrics Dr. Louis Roussos

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google