Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 1 Estimation of Item Difficulty Index Based on Item Response Theory for Computerized Adaptive Testing Authors ： Shu-Chen Cheng, Guan-Yu Chen

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology Introduction 2. Literature Reviews 3. Methods 4. Experiments and Results 5. Conclusions Outline

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 3 3 Computerized Adaptive Testing –Item Response Theory  Advantage: Personalized test, Shorter test length.  Shortcoming: The number of pre-test samples. IRT-1PL: 20 items, 200 testees (Wright & Stone, 1979) IRT-2PL: 30 items, 500 testees (Hulin et al., 1982) IRT-3PL: 60 items, 1000 testees (Hulin et al., 1982) ( There are 1,513 items in our item bank ！ ) 1. Introduction (1/2)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 1. Introduction (2/2) Test System = Item Bank + Item Selection  Item Difficulty Index  Answers Abnormal Rate  Dynamic Item Selection Strategy  Particle Swarm Optimization 4

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology Computerized Adaptive Testing 2.2 Item Difficulty Index 2.3 Item Response Theory 2. Literature Reviews

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 6 6 To select the item that its difficulty is most consistent with testee’s ability. To assess testee’s ability immediately. The difficulty of next item is affected by previous answer. 2.1 Computerized Adaptive Testing (1/2)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 7 7 To test for different abilities through dynamitic item selection strategy. –High ability testee  No too easy items. –Low ability testee  No too difficult items. A personalized test. 2.1 Computerized Adaptive Testing (2/2)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 2.2 Item Difficulty Index (1/2) Method 1 ： 8 P: Item difficulty. R: The number of correct answers. N: The number of total testees.

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 2.2 Item Difficulty Index (2/2) 9 Method 2 ： P: Item difficulty. P H : Correct rate of high score group. P L : Correct rate of low score group. (Generally take 25%, 27%, 33%, etc.)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 10 Item Response Theory (Lord, 1980) –To estimate testee’s ability, aptitude, or location of other continuous psychological interval by the information of their item responses. –Ability location  Item response (Psychometric theory) –In addition to the model of IRT, without any other information to describe the item responses. 2.3 Item Response Theory (1/2)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology Item Response Theory (2/2)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 12 Answers 1)Testees’ ability > Item difficulty index  Most testees are supposed to answer correctly. 2)Testees’ ability < Item difficulty index  Most testees are supposed to answer wrong. 3)Testees’ ability = Item difficulty index  The correct answer rate is 50%. 3. Methods (1/4)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 13 Answers Abnormal –Violations of any one of these above 3 assumptions among answers are answers abnormal.  1st group with wrong answers. (Testee’s ability > Item difficulty)  2nd group with correct answers. (Testee’s ability < Item difficulty)  3rd group, correct answer rate ≠ 0.5. (Testee’s ability = Item difficulty) 3. Methods (2/4)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 14 Answers Abnormal Rate T ： The number of correct answers. F ： The number of wrong answers. N ： The number of total testees. h ： 1st group (Testee’s ability > Item difficulty). l ： 2nd group (Testee’s ability < Item difficulty). e ： 3rd group (Testee’s ability = Item difficulty). 3. Methods (3/4)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology Methods (4/4) 15 Item Difficulty Difficulty j, letbe the smallest. : Item difficulty index of item i. Answers abnormal rate of item i with difficulty j. :

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology System Descriptions 4.2 Experiment Descriptions 4.3 Results and Discussions 4. Experiments and Results

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology System Descriptions (1/3)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology System Descriptions (2/3)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 4.1 System Descriptions (3/3) 19 PSO Dynamic Item Selection Strategy Item Difficulty Knowledge Weights Item Exposure Rate

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology Experiment Descriptions Method: Online test Item Bank: –Items: 1,513 –Initial Difficulty: 0.5 (9 levels, 0.1~0.9) Participants: –Students: 51 –Initial Ability: 0.2 (9 levels, 0.1~0.9) Periods: 6 weeks

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology Results and Discussions (1/3)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology Results and Discussions (2/3)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology Results and Discussions (3/3)

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology Conclusions Each test item is treated as independent, and the item difficulty can be estimated individually. Therefore, the item bank can be expanded easily at any time. The estimation based on the answers abnormal rate proposed in this study can estimate the item difficulty index quickly and reasonably without too many pre- test samples.

Intelligent System Lab. (iLab) Southern Taiwan University of Science and Technology 25 The End ~ Thanks for your attention!