1. A procedure for dimensionality analyses of response data from various test designs Jinming Zhang William Stout

2. Introduction  Dimension  Dimensional structure of the test (e.g., algebra and geometry)  statistical dimensional structure of response data  Incorporate both Judgments about test content and evidence from statistical analyses  Missing data: CAT & multistage testing  Missing item pair measurement

5. DETECT index

6. Modified DETECT index

7. Example 3 2 dim2 dim1 dim (additional dim) 1 dim (additional dim)  Stage 1 booklet: 2 dim & Stage 2 booklet 1: 2 dim, Stage 2 booklet 2: 1 dim (additional dim), Stage 2 booklet 3: 1 dim (additional dim).  Bridge item (in Stage 1)  E(1,2) and E(2,3), so E(1,3)  First-stage booklet should  measure all of the constructs/contents the whole test aim to measure, though it is unknown.  classify examinees into different proficiency levels

8.  Item 1 and 2 are measuring the same dimension 1. If E(1,2), E(2,3), and E(1,3)  D(P*) is maximized in P* partition if item 1 and 2 are in the same cluster, other than in other P 2. If only E(1,2) and E(2,3), and if item 2 is a bridge item  D(P*) is maximized in P* partition if item 1 and 2 are in the same cluster when item 2 is in the same cluster or not, other than in other P

10. Whether a test has an approximate simple structure or not

11. Discordance  Resulted from  Not a approximate simple structure  Inaccuracy of conditional covariance estimation  Given a partition of items,

13.  What made Dd(P*) and PropD(P*) large  Unidimensionality  Violation of approximate simple structure  Inaccuracy of conditional covariance estimation

14. polyDETECT  Obtain the composite theta score: unidimensinoal approximation & simple structure approach.  Use percentiles of composite theta scores as cut-off points in forming AHGs.  Between 25 to 100 in each group  Cross-validation:

15. polyDETECT  Evidence of multidimensionality

16. Condition 3 is hold or not?  Each sizable cluster contains at least one stage-1 item.  Exist at least one sizable cluster that does not contain any Stage-1 items, and all items in such cluster belong to the same booklet.  Exist at least one sizable cluster that does not contain any Stage-1 items, and all items in such cluster belong to at least two booklet.  Exist at least two sizable cluster that does not contain any Stage-1 items, items in each such cluster come from the same booklet but different clusters belong to different booklets.

17. Dealing with CAT data  Estimates of item parameters were obtained before conducting CAT. Why do dimensionality assessment on CAT data?  Sparse data set of CAT  100,000 responses are required at least  Item selection, item exposure control, content balance to satisfy the condition 3

18. Simulation study  M2PL  Each booklet has 30 items  Dimension: 1, 2, 3  Number of examinees: 750, 1500, 3000, 4000  Theta: MVN(0,sigma), correlation = 0.8  Cut-off points for low-, moderate-, and high- scoring group: 18  About 37.82% unestimable item pairs  Cross-validation  Replication: 100  Composite theta: use unidimensional IRT model

19. Results

20. Real data analyses  Missing values are large (55%~71%)

21. Real data analyses  Weak dimensionality (M value)  High PropD(P*) indicates a large amount of spurious information to form partition P*  Confirmatory analyses: D3 >D2  But DETECT tends to underestimate, so two- cluster partition solution may be preferred.

22. Real data analyses  High correlation indicates a weak degree of multidimensionality

23. Concluding remarks  Moderate violation of approximate simple structure is still hold