Item response modeling of paired comparison and ranking data

paired comparison and ranking data paired comparison – n(n-1)/2 pairs Ranking – n! – A special case of paired comparison when no intransitive pattern Thurstone’s model (1927) – Utility (property of item)

Ranking Y pair = 1 if ti – tk > 0 Design matrix: Utility distribution: – Case 3: – Case 5: Separate person parameter out of random error: – Note they think of loading parameter as item attributes (e.g., male actors versus female athletes)

Pairwise comparison Add intransitive error: Then Identification: – Origin and scale: N(0,I) for η – Rotation:  – Additions due to pairwise design: Fix loading parameters of a statement to 0. Fix one mean parameter of a statement to 0. Fix one unique variance of a statement to 1.

Identification: – At least n=5, 6, 7 for m = 1, 2, 3 (number of dimension). Why? – If not, require more constraints! What it is? Constrain All the covariance matrix (I have tried this!)

Thurstonian IRT model Recall Do substitution Take n=3 as example:

Parameter estimation MML may be infeasible because ICCs are conditionally dependent for Thurstone IRT model. Limited information method is applicable by using Mplus. But d.f. should be modified for ranking data:

Item characteristic function Recall Y* = Re-expressed as Note

Latent trait estimation, information functions, and reliability estimation Locally independence is violated! MAP Information function Reliability – 1. – 2.

Simulation studies To estimate Sample size: 200, 500, 1000 Item size: 6, 12 Equal or unequal variance:

Results

How "close" are the observed values to those which would be expected under the fitted model?

MAP

Vocational interest (pairwise comparison) Unrestricted thresholds: p=.046, RMSEA=.016 Equal w: p=.000, RMSEA=.025 Constrained thresholds: p=.000, RMSEA=.025 Reliability:.62 (theoretical);.43 (empirical) due to shrunken MAP Realistic Investigative Artistic Conventional Social Enterprising

Vocational interest (pairwise comparison)

Work motivation (ranking data) Chi-square fit index: p=.000, RMSEA=.062

Work motivation (ranking data) Reliability:.74 (theoretical);.76 (empirical)

Discussion Locally dependence – Using MCMC Discrimination: positively vs negatively worded Multiple traits Forced-choice design