Variability in the skills measured by tests of “reading comprehension across tests and across grade levels Dr. Joseph Torgesen Florida State University and Florida Center for Reading Research

The Florida Center for Reading Research recently conducted a study of the FCAT to answer several important questions: Among the most important questions were these: What are the reading, language, and cognitive abilities that are most important in explaining individual differences in performance on the FCAT at 3rd, 7th, and 10th grades? What kinds of skills and knowledge are particularly low in students who struggle on the FCAT?

The FCAT has some important characteristics It was specifically created to place high demands on vocabulary and reasoning/inferential skills “FCAT demands an in-depth understanding and application of information that is not typical of most standardized tests.” (Lessons Learned, 2002) Design specifications call for “application of skills in cognitively challenging situations.” Proportion of questions requiring “higher order” thinking skills increases from 30% in grade three to 70% in grade 10

The FCAT has some important characteristics The FCAT may also place special demands on reading fluency, as opposed to “labored accuracy” Passage length at different levels 3rd grade – 325 words 7th grade – 816 words 10th grade – 1008 words

How the study was conducted: Gave 2 hour battery of language, reading, nonverbal reasoning, and memory tests to approximately 200 children in each grade at 3 locations in the state Language – Wisc Vocab and Similarities Listening comprehension with FCAT passage Reading– Oral reading fluency, TOWRE, Gray Oral Reading Test The locations in which students were tested was Tallahassee, Tampa, and Ft. Lauderdale. We used a selection of schools that was roughly representative of the full SES range. The sample in this study is very similar in demographic characteristics of students (percent minority, percent free reduced lunch) to state averages. The language factor represents students broad knowledge of words and word meanings as well as their ability to engage in verbal reasoning The Reading factor is primarily oral reading fluency. Nonverbal reasoning is how well students can perform on reasoning tasks that involve thinking about matrixes and designs Working memory is a measure of “how many things a person can think about at one time. NV Reasoning – Wisc Matrix Reasoning, Block Design Working Memory– Listening span, Reading Span

Percent of variance accounted for Fluency 60 Verbal 55 Non Verbal Memory 50 47 40 3rd Grade Percent of variance accounted for 30 23 20 17 7 2 12 10 Individually Unique

What skills are particularly deficient in level 1 and level 2 readers in 3rd grade? Skill/ability FCAT Performance Level 1 2 3 4 5 WPM on FCAT 54 92 102 119 148 WPM on DIBELS 61 96 111 132 155 Fluency percentile 6th 32th 56th 78th 93rd Phonemic decoding 25th 45th 59th 74th 91st Verbal knowledge/ reasoning 42nd 59th 72nd 91st 98th

7th Grade 60 51 50 43 Percent of variance accounted for 40 30 22 20 10 Fluency 60 Verbal Non Verbal 43 51 22 5 Memory 50 40 7th Grade Percent of variance accounted for 30 20 8 10 2 10 Individually Unique

What skills are particularly deficient in level 1 and level 2 readers at 7th grade? Skill/ability FCAT Performance Level 1 2 3 4 5 WPM on FCAT 88 113 122 144 156 Fluency percentile 7th 25th 45th 82th 95th Phonemic decoding 27th 53rd 53rd 74th 84th Verbal knowledge/ reasoning 34th 45th 64th 88th 93rd

10th Grade 60 32 52 28 5 50 40 Percent of variance accounted for 30 20 Fluency 60 Verbal Non Verbal 32 52 28 5 Memory 50 40 10th Grade Percent of variance accounted for 30 20 7 15 2 10 Individually Unique

What skills are particularly deficient in level 1 and level 2 readers at 10th grade? Skill/ability FCAT Performance Level 1 2 3 4 5 WPM on FCAT 130 154 175 184 199 Fluency percentile 8th 30th 68th 87th 93rd Phonemic decoding 18th 27th 45th 56th 72nd Verbal knowledge/ reasoning 30th 60th 66th 84th 89th

A study of intensive, highly skilled intervention with 60 children who had severe reading disabilities Children were between 8 and 10 years of age Had been receiving special education services for an average of 16 months Nominated as worst readers: at least 1.5 S.D’s below grade level Average Word Attack=69, Word Identification=69, Verbal IQ=93 Randomly assigned to two instructional conditions that both taught “phonics” explicitly, but used different procedures with different emphasis Children in both conditions received 67.5 hours of one-on-one instruction, 2 hours a day for 8 weeks Children were followed for two years after the intervention was completed

R=.55 Differences in outcomes across measures of comprehension Pretest Posttest 2-year follow Accuracy Fluency Comp-WMRT Comp-Gray 74 71 83 73 89 75 91 86 91 73 95 88 Thus, we have evidence that the FCAT increasingly assesses “higher order thinking skills” as grade level increases. R=.55

A study of intensive interventions with 3rd and 5th grade students using four different methods Children were in 3rd or 5th grade Nominated by schools, and tested below the 30th percentile in word reading efficiency Average Word Attack=92, Word Identification=90, Verbal IQ=94 240 children at each grade level randomly assigned to 4 instructional conditions Children in both conditions received approx. 90 hours of one-on-three instruction, 1 hour a day, five days a week, for approximately 6 months

Differences in outcomes across measures of comprehension Approximately 240 3rd grade students – received 90 hours of intervention—post test scores 1 2 3 4 Word ID Fluency Comp-WMRT Comp-GRADE 95 83 99 89 95 85 102 93 95 85 100 92 98 87 104 96 Thus, we have evidence that the FCAT increasingly assesses “higher order thinking skills” as grade level increases.

45 children in each of two intervention conditions A prevention study with first grade students – and 1 year follow-up data in second grade Children were in 1st grade at intervention, second grade follow-up Identified as 18% most at risk for reading failure based on performance on knowledge of letter sounds, and phonemic awareness at beginning of first grade 45 children in each of two intervention conditions Children received 60 minute sessions 4 days a week from October through May – about 80 hours of instruction

R=.58 Differences in outcomes across measures of comprehension 90 second grade children-one year followup 1 2 Accuracy Fluency Comp-WMRT Comp-GRAY 95 92 97 95 98 R=.58 Thus, we have evidence that the FCAT increasingly assesses “higher order thinking skills” as grade level increases.