1. CAN INSTRUCTIONALLY INSENSITIVE ACCOUNTABILITY TESTS EVER EVALUATE EDUCATORS FAIRLY? W. James Popham University of California, Los Angeles Winter Conference Washington Educational Research Association and Office of Superintendent of Public Instruction, Seattle December 4,2008

2. In nations where students’ scores on “accountability” tests play a pivotal role in the evaluation of schools, it is assumed that students’ performances on such tests accurately reflect instructional quality. In nations where students’ scores on “accountability” tests play a pivotal role in the evaluation of schools, it is assumed that students’ performances on such tests accurately reflect instructional quality.

3. But what if students’ scores on educational accountability tests did not accurately reflect instructional quality? But what if students’ scores on educational accountability tests did not accurately reflect instructional quality?

4. A DEFINITION OF INSTRUCTIONAL SENSITIVITY A DEFINITION OF INSTRUCTIONAL SENSITIVITY The degree to which students’ performances on a test accurately reflect the quality of instruction specifically provided to promote students’ mastery of what is being assessed. The degree to which students’ performances on a test accurately reflect the quality of instruction specifically provided to promote students’ mastery of what is being assessed.

5. 1 2 3 4 5 6 7 8 9 10 Completely Insensitive Totally Sensitive A Continuum of Instructional Sensitivity Accountability tests, such as numerous assessments used in the U.S., differ in their ability to detect instructional quality.

6. WHY MIGHT A TEST ITEM BE INSTRUCTIONALLY INSENSITIVE? Alignment Leniency Alignment Leniency Excessive Easiness Excessive Easiness Excessive Difficulty Excessive Difficulty Confusion-Engendering Item Flaws Confusion-Engendering Item Flaws Socioeconomic Status (SES) Links Socioeconomic Status (SES) Links Academic Aptitude Links Academic Aptitude Links

7. ALIGNMENT LENIENCY Many items on accountability tests, when judged as to their alignment with the curricular aims they are supposed to be measuring, will be regarded as aligned with those aims (skills and/or knowledge) even if the items are only tangentially related to the curricular aim being assessed. Many items on accountability tests, when judged as to their alignment with the curricular aims they are supposed to be measuring, will be regarded as aligned with those aims (skills and/or knowledge) even if the items are only tangentially related to the curricular aim being assessed.

8. An Example of Lenient Alignment Item 23 Using the bus schedule on the adjacent page, if your purpose was to determine the shortest time to reach Boston from Denver on a Monday, on which bus should you begin your journey? A. Bus 214 B. Bus 197 C. Bus 110 D. Bus 202

9. Was the item aligned? If the curricular aim had been for students to be able to use appropriate functional texts such as train or bus schedules. If the curricular aim had been for students to be able to use appropriate functional texts such as train or bus schedules. If the curricular aim had been for students to be able to determine whether given functional texts would fulfill their purpose for using such texts. If the curricular aim had been for students to be able to determine whether given functional texts would fulfill their purpose for using such texts.

10. EXCESSIVE EASINESS If an item is so easy that even completely untaught students would answer it correctly, then the item can’t distinguish between well taught and poorly taught students. If an item is so easy that even completely untaught students would answer it correctly, then the item can’t distinguish between well taught and poorly taught students. E.g., How many letters are there in the word seven? E.g., How many letters are there in the word seven?

11. EXCESSIVE DIFFICULTY If an item is so difficult that even marvelously instructed students might not answer it correctly, then the item can’t distinguish between well taught and poorly taught students. If an item is so difficult that even marvelously instructed students might not answer it correctly, then the item can’t distinguish between well taught and poorly taught students. E.g., Without using your computer, what is the square root of 1,522,756? E.g., Without using your computer, what is the square root of 1,522,756?

12. ITEM FLAWS Items embodying serious deficits (e.g., ambiguities, garbled syntax, more than one correct answer, or no correct answer) will prevent well taught students from answering the item correctly, hence make it impossible for the item to accurately distinguish between effectively and ineffectively taught students. Items embodying serious deficits (e.g., ambiguities, garbled syntax, more than one correct answer, or no correct answer) will prevent well taught students from answering the item correctly, hence make it impossible for the item to accurately distinguish between effectively and ineffectively taught students.

13. SOCIOECONOMIC STATUS (SES) LINKS If an item gives a meaningful advantage to students from higher SES families, then the item will tend to measure what students bring to school rather than how well they are taught once they get there. If an item gives a meaningful advantage to students from higher SES families, then the item will tend to measure what students bring to school rather than how well they are taught once they get there.

14. A plant’s fruit always contains seeds. Which of the items below is not a fruit? A plant’s fruit always contains seeds. Which of the items below is not a fruit? A.orange B.pumpkin C.apple D.celery A 6th-Grade Science Item:

15. In which of the sentences below does the word field mean the same thing as in the sentence above? In which of the sentences below does the word field mean the same thing as in the sentence above? A.The shortstop knew how to field his position. B.We prepared the field by plowing it. C.What field do you plan to enter when you graduate? D.The nurse examined my field of vision. A 4th-Grade Reading Item: My father’s field is computer graphics.

16. ACADEMIC APTITUDE LINKS If an item gives a meaningful advantage to students who possess greater inherited quantitative, verbal, or spatial aptitudes, then the item will tend to measure what students bring to school rather than how well they are taught once they get there. If an item gives a meaningful advantage to students who possess greater inherited quantitative, verbal, or spatial aptitudes, then the item will tend to measure what students bring to school rather than how well they are taught once they get there.

17. If someone really wants to conserve resources, one good way to do so is to: If someone really wants to conserve resources, one good way to do so is to: A.leave lights on even if they are not needed. B.wash small loads instead of large loads in a clothes-washing machine. C.write on both sides of a piece of paper. D.place used newspapers in the garbage. A 6th-Grade Social Studies Item:

18. The secret number is inside the circle. It is also inside the square. It is NOT inside the triangle. Which of these is the secret number? The secret number is inside the circle. It is also inside the square. It is NOT inside the triangle. Which of these is the secret number? A. 2B. 3 C. 5 D. 7 A 3rd-Grade Mathematics Item: 52 4 7 6 1 3

19. A 4 th -Grade Mathematics Item: Which of the letters below, when folded in half, will have two parts that match exactly? Which of the letters below, when folded in half, will have two parts that match exactly? F(A)Z(B) S(C)B(D)

20. WHY MIGHT A TEST ITEM BE INSTRUCTIONALLY INSENSITIVE? Alignment Leniency Alignment Leniency Excessive Easiness Excessive Easiness Excessive Difficulty Excessive Difficulty Confusion-Engendering Item Flaws Confusion-Engendering Item Flaws Socioeconomic Status (SES) Links Socioeconomic Status (SES) Links Academic Aptitude Links Academic Aptitude Links

21. A LESSON TO BE LEARNED: When the measurement community became convinced that assessment bias in our high-stakes tests was threatening validity, we set out to (1) detect assessment bias and (2) reduce it. We were successful. We can be equally successful in coping with instructional insensitivity. When the measurement community became convinced that assessment bias in our high-stakes tests was threatening validity, we set out to (1) detect assessment bias and (2) reduce it. We were successful. We can be equally successful in coping with instructional insensitivity.

22. TWO STRATEGIES FOR DETERMINING INSTRUCTIONAL SENSITIVITY A Judgmental Strategy whereby seasoned, well trained educators supply item-by-item ratings using a rigorous item-evaluation rubric A Judgmental Strategy whereby seasoned, well trained educators supply item-by-item ratings using a rigorous item-evaluation rubric An Empirical Strategy contrasting per-item performances of (1) taught versus untaught students or (2) effectively taught versus ineffectively taught students An Empirical Strategy contrasting per-item performances of (1) taught versus untaught students or (2) effectively taught versus ineffectively taught students

23. JUDGMENTAL DETERMINATION OF AN ITEM’S INSTRUCTIONAL SENSITIVITY An Illustrative Review Question: An Illustrative Review Question: “If a teacher has provided reasonably effective instruction related to the objective measured by this item, is it likely a substantial majority of the teacher’s students will respond correctly to the item?” “If a teacher has provided reasonably effective instruction related to the objective measured by this item, is it likely a substantial majority of the teacher’s students will respond correctly to the item?”

24. EMPIRICAL DETERMINATION OF AN ITEM’S INSTRUCTIONAL SENSITIVITY Contrasting per-item performances of taught versus untaught students Contrasting per-item performances of taught versus untaught students Contrasting per-item performances of effectively taught versus ineffectively taught students Contrasting per-item performances of effectively taught versus ineffectively taught students

25. INSTRUCTIONAL INSENSITIVITY: UNFAIR AND HARMFUL When we allow educators’ quality to be determined on the basis of accountability tests incapable of performing that task, we are being profoundly unfair to those educators. When we allow educators’ quality to be determined on the basis of accountability tests incapable of performing that task, we are being profoundly unfair to those educators. Far worse, some wrongly evaluated, desperation-driven educators will engage in classroom practices that are educationally harmful to children. Far worse, some wrongly evaluated, desperation-driven educators will engage in classroom practices that are educationally harmful to children.

26. Presenter’s e-mail address: wpopham@ucla.edu Reactions or suggestions regarding this topic will be welcomed.