Presentation is loading. Please wait.

# Exploring new versions of DIMTEST for use with Polytomous Data Tan Li Louis Roussos Measured Progress July 24, 2009.

## Presentation on theme: "Exploring new versions of DIMTEST for use with Polytomous Data Tan Li Louis Roussos Measured Progress July 24, 2009."— Presentation transcript:

Exploring new versions of DIMTEST for use with Polytomous Data Tan Li Louis Roussos Measured Progress July 24, 2009

Outline Introduction Dimensionality Hypothesis Test Poly-DIMTEST Methods Poly-DIMTEST without AT2 Poly-NEWDIM Simulation Study Results & Conclusions Future Work

Introduction Dimensionality Hypothesis Test Important assumption for many IRT models Equating Scoring Scaling Calibration DIF analysis Hypothesis Test H 0 : d E = 1 vs. H 1 : d E > 1

Introduction Poly-DIMTEST ( Nandakumar, Yu, Li, & Stout, 1998 ) Hypothesis test H 0 : vs. H 1 : Split all the test items into three subtests: AT1, AT2 and PT The test statistic: Stand Error of CCOV comes from a complicated formula

Introduction Poly-DIMTEST Weaknesses Difficulty on finding and choosing AT2 items Not enough items left for PT

Methods Poly-DIMTEST without AT2 Based on dichotomous version of DIMTEST without AT2 (Stout, Froelich, & Gao, 2001) Steps AT and PT 1. Split all the test items into two subtests: AT and PT 2. Fit a unidimensional nonparamatric model to the original data by kernel smoothing take the place ofAT2 3. Simulate N samples from the model to take the place of AT2 The test statistic: Stand Error of CCOV comes from the same formula provided by Nandakumar, et al. (1998)

Methods Poly-NEWDIM Based on dichotomous version of NEWDIM (Seo & Roussos, 2009) Similar procedure with Poly-DIMTEST without AT2 The test statistic: Standard Error Standard Error comes from the Standard Deviation over the simulated samples

Simulation Study Dichotomous items All of the parameters were randomly generated from the distributions based on real data from a large multi-year pool of 729 grade 5 math items

Simulation Study Polytomous items All of the parameters were randomly generated from the distributions based on real data from a large multi-year pool of 729 grade 5 math items

Simulation Study Type I Error Study Power Study 2 dimensions simple structure

Simulation Study Factors 500 examinees and 1000 examinees 52 pts test and 32 pts test AT subtest 52pts test: 5 MC, 10 MC, 2 CR, and 5 CR items 32pts test: 3 MC, 6 MC, and 3 CR items 52 pts 32 pts Dich. 5232120322080 Poly. 0510130368

Results Type I Error for 52 points test Sample sizePD-NO-AT2PND 500 0.02040.0350 1000 0.01750.0298 Average 0.01900.0324 Type I Error for 32 points test Sample sizePD-NO-AT2PND 500 0.00640.0292 1000 0.00580.0350 Average 0.00610.0321 Power for 52 points test Sample SizePD-NO-AT2PND 500 77.6990.58 1000 89.1096.08 Average 83.4093.33 Power for 32 points test Sample SizePD-NO-AT2PND 500 54.8972.56 1000 68.8683.00 Average 61.8877.78

Results Type I Error – 52 pts test, 400 trials Poly-DIMTEST without AT2 D,P5 MC10 MC2 CR5CR 52,03.55.5 32,51.25305.75 12,102.250.501 0,13 01.75 Poly-NEWDIM D,P 5 MC10 MC2 CR5CR 52,05.757 32,53.751.53.7511.25 12,1020.251.50.75 0,13 2.25 500 Examinees 7 Poly-DIMTEST without AT2 D,P5 MC10 MC2 CR5CR 52,02.253 32,5110.58.75 12,102.250.2501 0,13 01 Poly-NEWDIM D,P 5 MC10 MC2 CR5CR 52,054.5 32,53.50.54.257.75 12,101.25021.75 0,13 32.25 1000 Examinees

Results Type I Error – 32 pts test, 400 trials Poly-DIMTEST without AT2 D,P 3 MC6 MC3 CR 32,012 20,310.50.75 8,6000.25 0,8 0.25 Poly-NEWDIM D,P3 MC6 MC3 CR 32,044.75 20,352.255 8,61.750.51.25 0,8 1.75 500 Examinees 7 Poly-DIMTEST without AT2 D,P 3 MC6 MC3 CR 32,00.251.5 20,310.251.25 8,6001 0,8 0 Poly-NEWDIM D,P 3 MC6 MC3 CR 32,066 20,33.7518 8,60.50.253.75 0,8 2.25 1000 Examinees

Results Power – 52 pts test,400 trials Poly-DIMTEST without AT2 D,P 5 MC10 MC2 CR5CR 52,06196.5 32,558.7591.562.599.75 12,105192.2558.5100 0,13 60.5100 Poly-NEWDIM D,P5 MC10 MC2 CR5CR 52,083.5100 32,573.2597.2594100 12,1063.759193.25100 0,13 91100 500 Examinees < 85 ≥85 1000 Examinees Poly-DIMTEST without AT2 D,P5 MC10 MC2 CR5CR 52,079.2599.5 32,573.2597.7585100 12,1068.259985100 0,13 82.25100 Poly-NEWDIM D,P5 MC10 MC2 CR5CR 52,093.25100 32,586.2599.599100 12,1081.2597.75 100 0,13 98.25100

Results Power – 32 pts test, 400 trials Poly-DIMTEST without AT2 D,P 3 MC6 MC3 CR 32,019.565.75 20,31756.7592.25 8,6115389.75 0,8 89 Poly-NEWDIM D,P 3 MC6 MC3 CR 32,049.7586 20,34475.598.5 8,63666.598.5 0,8 98.25 500 Examinees < 85 ≥85 Poly-DIMTEST without AT2 D,P3 MC6 MC3 CR 32,032.582.75 20,32974.7598.75 8,625.574.7598.5 0,8 98.75 Poly-NEWDIM D,P3 MC6 MC3 CR 32,065.2595.5 20,360.588100 8,654.7583.599.75 0,8 99.75 1000 Examinees

Conclusion Type I error study Conservative Type I error behavior Poly-NEWDIM performs closer to nominal (0.05). Power study Poly-NEWDIM has greater power than Poly- DIMTEST without AT2 Poly-NEWDIM provides adequate power for a variety of conditions.

Future Work More examinees Dimensionality structure Item parameter simulation models Develop a method to choose AT subtest for mixed MC and CR tests Real datasets Skewed ability distributions

Reference Nandakumar, R., Yu, F., Li, H., & Stout, W. (1998). Assessing Unidimensionality of Polytomous Data. Applied Psychological Measurement, 22, 99-115. Stout, W., Froelich, A., & Gao, F. (2001). Using Resampling Methods to Produce an Improved DIMTEST Procedure. Essays on item response theory, 357-375 Seo, M., & Roussos, L. (2009). Evaluation of DIMTEST Effect-Size Measure and Its Application. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego.

Acknowledgement Measured Progress Department of Psychometrics Dr. Louis Roussos

Download ppt "Exploring new versions of DIMTEST for use with Polytomous Data Tan Li Louis Roussos Measured Progress July 24, 2009."

Similar presentations

Ads by Google