2. Computerized Adaptive Testing: What is it and How Does it Work? Presented by Matthew Finkelman, Ph.D.

3. Goals of this session Learn about Computerized Adaptive Testing (CAT) Crash course in Item Response Theory (IRT) Combining CAT with IRT History, pros and cons of CAT Answer questions

4. Not to be confused with… Computerized Adaptive Testing: Not as cute, but far fewer hairballs.

5. PART I Introduction to CAT

6. Why should you care? There are already operational assessments that use CAT Some believe it will revolutionize classroom testing in the future Fascinating idea that speaks to potential of computers to have new uses in education Item Response Theory is all over testing now

7. OK, so what is CAT? A type of assessment where a question is displayed on a monitor Students use mouse to select answer Computer chooses next question based on previous responses Next question is displayed on monitor, or else test ends

8. A graphical representation Questions chosen depend on prior responses

9. Analogy: A Game of 20 Questions I am thinking of an object. You have 20 “yes-or-no” questions to figure it out. Would you write out all your questions ahead of time? 1) Is it an animal? 2) Is it a vegetable? 3) Is it blue? 4) Is it red? 5) Is it bigger than a car? 6) Etc.

10. 20 Questions, Continued Isn’t it more effective to base your next question on previous answers? 1) Is it an animal? NO. 2) Is it a vegetable? YES. 3) Is it commonly found in a salad? YES. 4) Is it green? NO. 5) Would Bugs Bunny eat it? YES.

11. Same principle used in CAT Computer keeps track of each student’s pattern of responses so far As test progresses, learn more about individual student Choose next question (item) to get maximal info about that particular student’s level of ability Purpose of assessment: Get best possible information about students

12. Some items are more informative than others? Sure! Some items are easier than others: 2 + 2 vs. 54389 + 34697 Some items are more relevant than others: 3 + 7 vs. Academy Awards question Some items are better at discerning proficient students from those who need improvement

13. Which is most informative? Suppose we have only 2 types of students: “Advanced” and “Beginning” Use the test to classify each student Which item below is the best for this purpose? ItemP(Correct|Advanced)P(Correct|Beginning) 152% 275%34% 3100%0%

14. Item 3 is the best Item 1 is completely useless Item 2 gives some information Item 3 is all you need! ItemP(Correct|Advanced)P(Correct|Beginning) 152% 275%34% 3100%0%

15. But wait… Wouldn’t we choose Item 3 for ALL students? If so, why customize a test for an individual student? Answer: For some students, Item A is more informative. For others, Item B is more informative.

16. When is Item A more informative than Item B? Item A: 2 + 2 Item B: (34 + 68) / 2 If you’ve answered many difficult items correctly, Item A is waste of time If you’ve answered many easy items incorrectly, Item B is too hard Thus, give Item B to high-performing students, Item A to low-performing students

17. Isn’t that unfair? It seems like CAT penalizes students for performing well at start If we give different items to different students, how can we compare their performances? The above question arises whether we use CAT or not Item Response Theory to the rescue!

18. Summary of Part I CAT customizes assessment based on previous responses, as in 20 Questions Certain items more informative than others For some students, Item A is more informative; for others, Item B is When give different items to different students, need way to relate student performances (Item Response Theory)

19. PART II Crash Course in Item Response Theory

20. Item Response Theory (IRT) Quantifies the relation between examinees and test items For each item, gives probability of correct response by ability level Provides a means for estimating ability of examinees, describing characteristics of items Places examinees on common scale when they have taken different items

21. The IRT Model: One item

22. Different items have different curves

23. Where did those curves come from? In IRT, ability is denoted by θ Probability of a correct response is Each item has its own values of a, b, and c. We know them from “pre-testing” a is the “discrimination”: Related to the slope b is the “difficulty”: Harder item, higher b c is the “guessing parameter”: Chance of lucky guess

24. Effect of the a parameter Larger a increases the slope in the middle All curves shown have equal b and c parameters

25. Effect of the b parameter Larger b means harder item All curves shown have equal a and c parameters

26. Effect of the c parameter c is the left asymptote All curves shown have equal a and b parameters

27. Wait a minute What do you mean by a student with an ability of 1.0? Does an ability of 0.0 mean that a student has NO ability? What if my student has a reading ability of -1.2? What in the world does that mean???

28. The ability scale Ability is on an arbitrary scale that just happens to be centered around 0.0 We use arbitrary scales all the time: –Fahrenheit –Celsius –Decibels Nevertheless, need more “user-friendly” reporting: “scaled” scores on conventional scale like 200-300

29. Giving a score for each student First assign an ability (θ) value to each student (say, -3 to 3) Student is given the value of θ that is most consistent with his/her responses The better he/she does on the test, the higher the value of θ that he/she receives Computer converts the θ score to a scaled score Report final score!

30. Assigning scores Set of answers: (C,C,I,C,C,I,I,C,C,C,I,C,C) We know which items were taken by each student: a, b, c parameters If Student 1’s items were easier than Student 2’s, take into account through item parameters Student 1: θ = 1.25, scaled score = 290 Student 2: θ = 0.65, scaled score = 268 Can compare students who took different items!!!

31. Summary of Part II If you didn’t get all that, don’t worry Just remember: –In IRT, different items have different curves (depending on a, b, c parameters) –IRT allows us to give scores on the same scale, even when students take different items These features critical in CAT So how do we choose which items to give?

32. PART III Combining CAT with IRT

33. CAT Reminder CAT customizes assessment based on previous responses For some students, Item A is more informative; for others, Item B is With IRT, it’s OK to give different items to different students

34. Which item would you choose next? PREVIOUS RESPONSES: 10 + 19 = ? Answered correctly. 27 + 38 = ? Answered incorrectly. 12 + 26 = ? Answered incorrectly. POSSIBLE ITEMS TO GIVE NEXT: 18 + 9 = ? 13 + 17 = ? 14 + 20 = ?

35. Item selection to match ability/difficulty Want to give items appropriate to ability 2 + 2 is not informative for high-performing students; (34 + 68) / 2 is not informative for low- performing students Student has taken 10 items, awaits 11th Classic approach: Give item whose difficulty (b) is closest to current ability estimate (θ)

36. Which item is better for θ = -1.2? Ability of a student Easier item Harder item

37. More complex item selection Previous method: Match difficulty to ability This criterion only uses b parameter and θ Recall that a parameter is related to slope, c is guessing parameter Shouldn’t we consider those when choosing next item?

38. Another item selection method Ideal item: Low value of c; value of b close to θ; high value of a “Fisher Information” combines these factors into a single number Choose item with highest Fisher Info

39. Game: Which item would you choose? Suppose our current estimate of θ is 0.6 Itemabc 10.8-0.20.25 21.10.40.15 31.02.20.18

40. Results If matching ability (estimate = 0.6) with difficulty, we would give Item 2 If using Fisher Info, we would give Item 2 Itemabc 10.8-0.20.25 21.10.40.15 31.02.20.18

41. Round 2 Suppose our current estimate of θ is 0.7 Itemabc 11.30.90.20 21.10.60.22 30.8-0.10.10

42. Round 2 Results If matching ability (estimate = 0.7) with difficulty, we would give Item 2 If using Fisher Info, we would give Item 1 Itemabc 11.30.90.20 21.10.60.22 30.8-0.10.10

43. Summary of Part III Tailor items to be most informative about individual student’s ability Do this by combining CAT with IRT One method: Match difficulty with current estimate of θ Another method: Take all parameters into account via Fisher Info

44. PART IV Historical and Practical Considerations

45. Brief history of CAT Flexilevel testing: The “Choose Your Own Adventure” of assessment Multi-stage tests: Routing test followed by second (tailored) stage of testing “Full-blown” CAT studied heavily in 70’s; item selection methods proposed CAT has been used in real testing

46. Problem: Content Balance In testing, must balance content (e.g., Math test of algebra, geometry, number sense) What if all your most informative items come from the same content strand? In practice, dozens of constraints for each CAT: Content, topics, enemies list, etc. CAT solution: Pick most informative item among those “in play”

47. Problem: Test security CAT administered on multiple occasions Different students, different items; however, some items more popular than others Person A takes exam, memorizes items, tells Person B. Person B takes exam, benefits from Person A’s information CAT solution: Limit the amount each item can be administered

48. Problems persisted CAT lovers were none too happy about this

49. CAT “Pros” Convenient administration Immediate scoring Items maximally informative: Exams just as accurate, with shorter tests Items at correct level: High-performing students not bored, low-performing students not overwhelmed

50. CAT “Cons” Limited by technology Potential bias versus students with less computer experience Content balance less exact than paper-and- pencil testing Test security

51. Use of CAT in diagnostic testing Want to use CAT where “pros” are in effect but “cons” are not Diagnostic testing: Give each student a list of strengths & weaknesses, helping teachers focus instruction Content “naturally” balanced Low-stakes, so test security less of an issue Still reap benefits of adaptive testing

52. Final summary Introduction to CAT: Benefits of giving different items to different students Crash course in IRT Using IRT to select items in a CAT History, pros and cons of CAT How should CAT be used in the future?

53. It’s all about student learning. Period. www.measuredprogress.org