1. 1 NCES National Forum Stats-WDC 2009 NESAC Presentation July 28, 2009 Al Larson, Ph.D., LEA Connecticut Longitudinal Data System and Effective Use of Data at the LEA Level

2. 2 Purpose of High Stake Tests is to Determine Ability Estimates Unidimensional Tests (IRT Models) are Designed for Policy Makers Summative Assessments Advanced Goal Proficient Basic Below Basic High (400) Low (100) Scale ScoreCut Scores and Labels Vertical Scale Scores measure “growth” (not proficiency levels)

3. 3 Math Strand 1: Place Value (about 4 types of questions) Math Strand 8: Computation with fractions and integers Need a finer “grain size” (NAEP and ETS) To be more diagnostic, state’s report strands

4. 4 Reading: Critical Thinking Reading Strands (Based on NAEP Contexts) Strand A:Forming a General Understanding Strand B:Developing Interpretation Strand C:Making Reader/Text Connections Strand D:Examining the Content and Structure Strong use of inference … no literal questions Connecticut: 60% to 65% p-value NAEP study by Zwick, 1987; and Strand raw score reported to teachers and administrators (without basic assessment literacy) Need a finer “grain size”

5. 5 LDS at LEA Over 300 data files linked by LEA ID number ID -------- -------- -------- -------- -------- -------- -------- -------- -------- SPSS Programming Merge by ID if …then … do if … compute …sort … print … report …………….

6. 6

7. 7 Red = bad Green = good

8. 8 For this student, our assessments have failed to tell us “what to do next” in a timely manner. We need to design tests for a new purpose: to help teachers and students identify cognitive errors to help guide current instruction (formative assessment).

9. 9 LEA Assessment Purpose ( finer grain size): Develop a district-wide assessment system that is meaningful to teachers in Math and Reading for grades 2-9 (3 to 4 or 5 administrations per year) Similar to Cognitive Diagnostic Assessments, multiple-choice items were constructed by designing foils/distracators that mimic typical student cognitive processing errors (teacher input: add constructed response) Utilize the error vocabulary of each domain to report error descriptions teachers understand The transition from report data to instruction, is more difficult in reading than math …

10. 10 1 + 1 = 24 The Cognitive Task MC Cognitive Task: evaluation of the differences between foils A 1/6 B 2/4 C 2/6 D 3/4 E 1/8 ?

11. 11 The “Effective Use of Data” is the Error Description: “adding both numerator and denominator” or “did not complete last step of multi-step problem” (Diagnostic Reports within 3 to 5 days) 1 + 1 = 2 246 The Cognitive Task and Error Description in Math MC Cognitive Task: evaluation of the differences between foils A 1/6 B 2/4 C 2/6 D 3/4 E 1/8 ? What is meaningful to teachers: a finer “grain size” in a timely manner

12. 12 #8 = ' ' answer8='absent or left blank'. /* strand 3 'Equivalent Fract, Decimal & Percents'. #8 = ‘A’ answer8='error: word prob; chose 1:6 vs. 1:3'. #8 = ‘B’ answer8='error: word prob; reversed ratio, chose 3:1 vs 1:3'. #8 = ‘C’ answer8='correct: word prob; 3:9 is the same as 1/3'. #8 = ‘D’ answer8='error: word prob; chose 3:12 (1:4) ratio vs. 1:3'. #8 = ‘E’ answer8='error: word prob; chose 2:3 ratio vs. 1:3'. #9 = ' ' answer9='absent or left blank'. /* strand 4 'Order, Magnitude, and Rounding of Numbers'. #9 = ‘A’ answer9='error: ordering from table: selected 3rd place'. #9 = ‘B’ answer9='error: ordering from table: selected 1st place'. #9 = ‘C’ answer9='error: ordering from table: selected 2nd place'. #9 = ‘D’ answer9='correct: ordering from table: found 4 th place ordering from G to L'. #9 = ‘E’ answer9='error: ordering from table: selected 5th place'. #10 = ' ' answer10='absent or left blank'. /* strand 5 'Models for Operations (one item)'. #10 = ‘A’ answer10='correct: word prob; chose correct number sentence ((6+4)X$5.00) for situation'. #10 = ‘B’ answer10='error: word prob; divided instead of multiplying'. #10 = ‘C’ answer10='error: word prob; divided instead of multiplying'. #10 = ‘D’ answer10='error: word prob; added all data vs. adding 6 & 4, then multiplying'. #10 = ‘E’ answer10='error: word prob; subtracted instead of multiplying'. A Finer Grain Size: Sample of other Cognitive Error Descriptions in Math About 4 times a year x 4 foils x 45 items x 7 grades = 5,040 error descriptions

13. 13 Reading is Different from Math  You can “see” math errors … but not reading inferential thinking errors  Math teachers are trained in an error vocabulary that is aligned with foil misconceptions … reading literature emphasize strategies, not errors  Math methods and materials (text books) are very similar in both format and rigor to state and federal high stakes tests …

14. 14 Disconnect: Testing vs. Teacher Training/Experience High Stakes Reading Tests are Inferential  Multiple Choice Items are difficult (p-values.3 to.7) Teacher Experience with Purchased Materials  too few multiple-choice and too easy (p-values.7 to.9)  too literal  current teaching methods emphasize constructed-response items that are often too accepting, and without a rubric  no assessment literacy Teacher Training and Vocabulary  Metacognition and fix-up strategies: look back to clarify, predict, author's purpose, main idea, activate background knowledge, etc. (Based upon Literature and NAEP contexts)

15. 15 Cognitive Model of Task Performance for Reading Comprehension Multiple-choice Items High Scoring versus Low Scoring Students Motivated and will spend time and effort to:  Be metacognitively aware;  “look back” to clarify/re-read; and  Evaluate differences between foils  Reading comprehension of the passage  Reading comprehension of the item stem  Reading comprehension of each foil  Sometimes just the nuance of one word in a foil (SAT, ACT, GRE, etc.) (identified error: “X”) (identified error: “T”) (identified error: “R”)

16. 16 Reading Errors that are Meaningful to Teachers CODEERROR (X)No-Support (“answer grabbing”) (T)Text Matching (“look back” for a literal answer) (R)Related (good but not the “BEST” = “tricky”) (L)Looks Like (O)Opposite (A)Anaphoric Pronoun Referent (s)Skipped Items Each EIa* foil is coded (a finer grain size): * Presented at 2009 AERA Convention under name of Error Identification assessments (EIa)

17. 17 Error Identification assessments (EIa) Text Matching and No-Support Foils Exhibit 1 (Teacher Copy) JOHN ROCK Item 65Why did the author write paragraph 4? T C T X A. To show how he became a dentist. B. To show John was a teacher. C. To show John was a hard worker who wanted to help black people. D. To show John was the first black lawyer to be recognized by the Supreme Court. E. To show John moved to the South during the Civil War. (he was a “dentist” in paragraph 3) (he was a “teacher” in paragraph 2) (inference) (this was described in paragraph 5) (he did not move to the South) Error Codes Foils Comment

18. 18 Error Identification assessments (EIa) The Carefully Crafted … Related Foil: (SAT, ACT, GRE …) requires students to evaluate subtle differences between foils Exhibit 2 (Teacher Copy) CLIMBING HIGH 67Which statement BEST describes what the passage is MOSTLY about? R C R T A. Annie was a woman who took many risks. B. Annie was a person who wasn’t afraid to follow her dreams. C. Women can be mountain climbers. D. Most mountains can be climbed. E. Most people never climb mountains. (she did take risks) (implied, best answer) (she did) (text matching) The related foil, as a constructed-response, would be an acceptable summative answer (critical thinking)

19. 19 Error Identification assessments (EIa) Sample Summary Error Identification Report to a Teacher (Students need to explain their reasoning) Summative Levels … and … Formative Diagnostics

20. 20 Error Identification assessments (EIa) The Assessment is Consumed for Instruction EIa test items and foils are used as instructional aids Lesson plans are in development Teachers conference with students:  an “internal view” with “retroactive verbal reports” (Leighton & Gierl; Norris; Gorin; 2007) ; “think alouds” (Davey, 1983) ;  Help students get involved in their own learning by making their thinking visible to themselves, peers and teachers; (Black & Wiliam, et al, 1998; Stiggins, et al, 2004, 2006); and  For the teaching of critical thinking (inference) and understanding of ideas in the text [foils] (Wells, 2000; Block, Gambrell & Pressley, 2002).

21. 21 Error Identification assessments (EIa) Teacher Opinion of EIa:  I use them to see what they are thinking … which errors seem to be used consistently - what they are “tricked on”. I turn it into a game-type activity: students vs. teacher (reading teacher, grades 2-5).  If students are aware of the errors they make, determined students will change their behavior both in reading and testing (classroom teacher, grade 5).  They now don’t grab the first answer that they connect to, they take more time to evaluate and critique each choice (reading teacher, unknown grade).  No support errors identifies students who clearly can't read on grade level or aren't taking it seriously (classroom teacher, grade 4).

22. 22 Error Identification assessments (EIa) Teacher Opinion of EIa:  I meet with students in small groups … Keeping the test booklets and handing them back to students is helpful in discussing why they chose their answers and it makes them accountable for their choices (classroom teacher, grade 4).  Understanding their misconceptions is one thing – getting them to change the misconception is the difficult part (classroom teacher, grade 7).  As we are working on it, [EIa post-conferencing] often students will “get” it when it is a clear error. The related [versus the] correct answer does not come as easily during the explanation (classroom teacher, grade 4). (Related is an important foil for “ability estimates” or critical thinking, some student’s will need more reading experience and scaffolding)

23. 23 With a combination of: EIa reports, items and teacher conferencing with students, teachers diagnose student misconceptions and provide scaffolding during repeated critical thinking activities. (Instructional Utilization of Diagnostic Data) Summary: EIa with Teachers for Instruction Error Identification assessments (EIa) EIaStudent Teacher

24. 24 Error Identification assessments (EIa) Reliability, Validity and Prediction Alpha Reliability Grades 2 to 9 2009 Beginning-Year 2009 Mid-Year 2008 End-Year Math.90 to.93.90 to.92.88 to.94 Readin g.84 to.88.80 to.86.84 to.89 All 60 reading comprehension passages are original Concurrent Validity Grades 3 to 9 2008 Beginning-Year 2008 Mid-Year Math.82 to.92.82 to.91 Reading.76 to.86.79 to.82 Prediction: Multiple R.85 to.93 in Reading

25. 25 Effective Use of Data: Matched Average Vertical Scale Score Growth in Reading Across Grades 4 to 8; F=16.6 P<.000 2008-2009 Instructional Utilization of Diagnostic Data 2345 N =345N = 913N = 855N = 363 lowhigh

26. 26 Effective Use of Data: Matched Average Vertical Scale Score Growth in Reading 2008-2009 Instructional Unitization of Diagnostic Data Gr 4 and 5 Gr 6 to 8 2 345 lowhigh