Presentation on theme: "Fit of Ideal-point and Dominance IRT Models to Simulated Data Chenwei Liao and Alan D Mead Illinois Institute of Technology."— Presentation transcript:
Fit of Ideal-point and Dominance IRT Models to Simulated Data Chenwei Liao and Alan D Mead Illinois Institute of Technology
Outline Background and Objective Hypotheses and Methods Results Discussions
Background Personality Used in personnel selection - Incremental validity to predict job performance beyond cognitive ability (Barrick & Mount, 1991; Ones et al, 1993) - Less adverse impact (Feingold, 1994; Hough, 1996; Ones et al, 1993). Model-data-fit - Need to calibrate personality traits - Use IRT models - Degree of fit depends on data structure
Background (cont.) Item response processes – thinking of data structure IRT models and item response processes: 1) Traditional dominance IRT models: - high trait - high probability of endorsing 2) Ideal-point IRT models - similar item & trait – high probability of endorsing
Background (cont.) Dominance Model IRF: - x: Theta (trait level) - y: Probability of endorsing Ideal-point Model IRF: - x: distance between person trait and item extremity - y: Probability of endorsing
Background (cont.) Chernyshenko et al, (2001) - Traditional dominance IRT models have failed. Suggest to look at item response processes and Ideal-point IRT models Stark et al. (2006) - Ideal-point IRT models: as good or better fit to personality items than do dominance IRT models Chernyshenko et al. (2007) - Ideal-point IRT method: more advantageous than dominance IRT and CTT in scale development in terms of model-data-fit
Limitation of previous studies and objective of current study Limitation of previous studies - Unknown item response processes! Objective of current study 1) Investigate model-data-fit by utilizing simulation with known item response processes 2) Test the assumption that the best fit model represents data underlying structure of response processes
Models Dominance: - Samejima ’ s Graded Response Model (SGRM); Ideal Point: - General Graded Unfolding Model (GGUM). Larger sample and longer test were said to be related to a better fit (Hulin et al, 1982; De la Torre et al, 2006).
Hypotheses Generating models H1: Data generated by an ideal point model will be best fit by an ideal-point model and data generated by a dominance model will be best fit by a dominance model. H2: The ideal point model will fit the dominance data better than the dominance model will fit the ideal-point data. H3: The ideal-point model will fit the mixture data better than the dominance model.
Hypotheses (cont.) Sample Sizes H4: All models will fit better in larger samples. H5: The GGUM model will fit relatively worse in smaller samples, as compared to simpler, dominance models. Test Lengths H6: The GGUM model will fit relatively worse for very short tests, as compared to longer tests.
Datasets Self-Control Scale from the 16PF Procedure: 1) Calibrate 16PF data to get item parameters - SGRM: PARSCALE4.1; GGUM: GGUM2004. 2) Generate simulated data: - models: ideal point/dominance/mixed; - sample size: 300, 2000; - test length: 10, 37; - 50 replications;
Model-Data-Fit Cross validation ratio: each item in each condition Only singles – simulation study assures unidimensionality assumption Smaller value – better fit Frequencies of ratios were tallied into 6 groups: very small ( =5).
Results overview ConditionBest fitting model Dominance data generationGGUM Ideal point data generationGGUM Mixed data generationGGUM Small Sample (N=300)GGUM Large Sample (N=2000)GGUM Short Test (n=10)GGUM Long Test (n=37)GGUM
Discussion (1) “ GGUM fits better ” - Confirm previous findings. - However, because regardless of the underlying response process, GGUM fits better than SGRM, it does not demonstrate that the response process or IRF/ORF is non-monotone. The previous assumption does not hold true. - Possible reason: Software (PARSCALE & GGUM) manifest models differently Better fit in small samples, especially for SGRM - Explanation: chi-square is sensitive to sample size
Discussion (2) Examine similarities of the theta metrics - Negative correlation between theta estimates from GGUM and those from SGRM TRUESGRMGGUM TRUE1.000 SGRM0.9281.000 GGUM-0.923-0.9951.000
Discussion (3) Scaling issue GGUM: - Reverse the estimate - Add a constant in scaling