1. Neural test theory model for graded response data SHOJIMA Kojiro The National Center for University Entrance Examinations, Japan shojima@rd.dnc.ac.jp 1

2. Weighing machine –A 1 weighs 73 kg –f W (A 1 )=73 f W (A 1 )≠74 f W (A 1 )≠72 Academic test –B 1 scores 73 points –f T (B 1 )=73 f T (B 1 )≠74 ? f T (B 1 )≠72 ? 2 Accuracy of tests

3. Discriminating ability of tests Weighing machine –A 1 weighs 73 kg –A 2 weighs 75 kg f W (A 1 )<f W (A 2 ) Academic test –B 1 scores 73 points –B 2 scores 75 points f T (B 1 )<f T (B 2 ) ? 3

4. Resolving ability of tests Weighing machine –A 1 weighs 73 kg –A 2 weighs 75 kg –A 3 weighs... Academic test –B 1 scores 73 points –B 2 scores 75 points –B 3 scores... 4 kg T

5. Neural Test Theory (NTT) Academic tests are an important public tool Precise measurements are difficult –10% measurement error Tests are at best capable of classifying academic ability into 5–20 levels Neural test theory (NTT) –Shojima, K. (2009) Neural test theory. K. Shigemasu et al. (Eds.) New Trends in Psychometrics, Universal Academy Press, Inc., pp. 417-426. –Test theory that uses the mechanism of a self-organizing map (SOM; Kohonen, 1995) –Latent scale is ordinal 5

6. 6 Graded evaluation ↓ Accountability ↓ Qualification test For Qualifying Tests Ordinal academic ability evaluation scale based on Neural Test Theory Continuous academic ability evaluation scale based on IRT or CTT It is difficult to explain the relationship between scores and abilities because individual abilities also change continuously Because the individual abilities also change in stages, it is easy to explain the relationship between scores and abilities. This increases the test ’ s accountability.

7. Statistical Learning Procedure in NTT ・ For (t=1; t ≤ T; t = t + 1) ・ U (t) ←Randomly sort row vectors of U ・ For (h=1; h ≤ N; h = h + 1) ・ Obtain z h (t) from u h (t) ・ Select winner rank for u h (t) ・ Obtain V (t,h) by updating V (t,h−1) ・ V (t,N) ←V (t+1,0) Point 1 Point 2 7

8. NTT Mechanism 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 1 1 0 1 0 1 0 0 1 Latent rank scale Number of i tems Response Point 1Point 2 Point 1Point 2 8

9. Point 1: Winner Rank Selection The least squares method can also be used. Bayes ML Likelihood 9

10. Point 2: Update the rank reference matrix The nodes of the ranks nearer to the winner are updated to become closer to the input data h: tension α: size of tension σ: region size of learning propagation 10

11. Analysis Example A geography test N5000 n35 Median17 Max35 Min2 Range33 Mean16.911 Sd4.976 Skew0.313 Kurt-0.074 Alpha0.704 11

12. Fit Indices ML, Q=10ML, Q=5 Useful for determining the number of latent ranks 12

13. Item Reference Profiles Monotonic increasing constraint can be imposed 13

14. Test Reference Profile (TRP) Weakly ordinal alignment condition –TRP increases monotonically, but not all IRPs increase monotonically Strongly ordinal alignment condition –All IRPs increase monotonically  TRP also increases monotonically For the latent scale to be an ordinal scale, it must at least satisfy the weakly ordinal alignment condition (WOAC). Weighted sum of IRPs Expected value of each latent rank 14

15. Rank Membership Profile (RMP) Posterior distribution of the latent rank to which each examinee belongs RMP 15

16. Examples of RMP 16

17. Extended Models Graded Neural Test Model (RN07-03) –NTT model for ordinal polytomous data Nominal Neural Test Model (RN07-21) –NTT model for nominal polytomous data Continuous Neural Test Model Multidimensional Neural Test Model 17

18. Graded NTT Model Boundary Category Reference Profiles

19. Graded NTT Model Item Category Reference Profile

20. Nominal NTT Model Item Category Reference Profile * Correct selection, x Combined categories selected less than 10 ％ of the time

21. Website http://www.rd.dnc.ac.jp/~shojima/ntt/index.htm Software –EasyNTT By Prof. Kumagai (Niigata Univ.) –Neutet By Prof. Hashimoto (NCUEE) –Exametrika By Shojima (NCUEE) 21

22. Demonstration of Exametrika 22

23. Can-Do Chart (Example) 23