The Problem of Comprehension: A Potential Barrier to Reading Curriculum-Based Measurement as a Progress Monitoring and Screening Instrument Mark R. Shinn, Ph.D. Professor and School Psychology Program National Louis University, Skokie, IL markshinn@icloud.com http://markshinn.org February 21st, 2013

Editor and Contributor to 2 Major Texts on CBM My Area of Expertise Editor and Contributor to 2 Major Texts on CBM 1 of 6 members of Technical Review Panel, National Center for Student Progress Monitoring, USDE/OSEP 2003-2007 Author of More than 75 Refereed Journal Articles and Book Chapters on the Topic of CBM, Progress Monitoring, and Screening

Disclosure Disclosure Mark R. Shinn, Ph.D. Serves as a Paid Consultant for Pearson Assessment for their AIMSweb product that provides CBM assessment materials and organizes and report the information from 3 tiers, including RTI. He provides technical support and training. Mark R. Shinn, Ph.D. Serves as a Consultant for Cambium/Voyager/Sopris for their Vmath product, a remedial mathematics intervention but has no financial interests. He helped them develop their progress monitoring system. Mark R. Shinn, Ph.D. Serves as a Consultant for McGraw-Hill Publishing for their Jamestown Reading Navigator (JRN) product and receives royalties.He helped them develop their progress monitoring system. Mark R. Shinn, Ph.D. Serves as a Member of the National Advisory Board for the CORE (Consortium on Reaching Excellence) and receives a stipend for participation. He provides training and product development advice.

Learning to Read is Critical for School (and Life) Success

And the Number 1 Most Powerful TEACHING Variable Frequent Formative Assessment Like Weekly Use of Reading Curriculum-Based Measurement (R-CBM) is Among our Most Powerful Tools! And the Number 1 Most Powerful TEACHING Variable Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge.

BUT... What about comprehension? What about “word callers?” I know this kid who...reads “fluently” but doesn’t understand a THING they read...

Presentation is Based on the Following White Paper in PRODUCTION Will Be Available in pdf format iBook format A “glossy” and official Pearson version will be finished soon and sent to you. Shinn, M.R. (2013). The problem of comprehension: A potential barrier to Reading Curriculum-Based Measurement (R-CBM) as a progress monitoring and screening instrument. Minneapolis, MN: Pearson Assessment.

References on CBM Criterion Related Validity Deno, S. L., Marston, D., Shinn, M. R., & Tindal, G. (1983). Oral reading fluency: A simple datum for scaling reading disability. Topics in Learning and Learning Disability, 2, 53-59. Fuchs, L. S., Fuchs, D., Hosp, M. K., & Jenkins, J. R. (2001). Oral reading fluency as an indicator of reading competence: A theoretical, empirical, and historical analysis. Scientific Studies of Reading, 5, 239-256. Fuchs, L. S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9, 20-28. Hamilton, C., & Shinn, M. R. (2003). Characteristics of word callers:  An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240. Miura Wayman, M., Wallace, T., Ives Wiley, H., Ticha, R., & Espin, C. (2007). Literature synthesis on curriculum-based measurement in reading. The Journal of Special Education, 41, 85-120. Shinn, M. R., Good, R. H., Knutson, N., Tilly, W. D., & Collins, V. (1992). Curriculum-Based reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21, 458-478.

Accessing Reading Materials markshinn.org Click on the Downloads for Professionals Icon Click on the Presentations and Handouts Folder Click on AIMSweb Webinar R-CBM and the Problem of Comprehension (Sponsored by Pearson) 2013 Folder 10

Big Ideas When students read passages aloud for 1 minute, we are obtaining a holistic assessment of general reading skills. A strong empirical relation has been demonstrated between R-CBM and measures of general reading and reading comprehension (RC). Persuasive empirical data are not always persuasive of teacher opinion. General reading skills are necessary, but not sufficient for comprehension. RC is not a single “thing."  Judgments are based on: What a student is expected to read. How the student will be assessed. RC is influenced by a number of important variables, including General reading ability Language, especially vocabulary and familiarity with cultural idioms Knowledge, especially content knowledge and knowledge of the world Metacognitive strategies, including motivation and interest

2 Other Big Ideas We Can’t Discuss Today If educators are concerned about specific student’s RC, then we must have a systematic assessment process to address these concerns, starting with ruling out general reading skill deficits, but also including interviews, observations, and diagnostic assessment. See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 671-698). Bethesda, MD: National Association of School Psychologists. If students have low general reading skills, RC strategy intervention should be time limited and the use of such strategies should be actively taught, expected, supported, and rewarded in Tier 1, general education and/or content area classes. See: Willingham, D.T. (2006/07). The usefulness of Brief instruction in reading comprehension strategies. American Educator, Winter, 39-50.

2 Other Big Ideas We Can’t Discuss Today If educators are concerned about specific student’s RC, then we must have a systematic assessment process to address these concerns, starting with ruling out general reading skill deficits, but also including interviews, observations, and diagnostic assessment. See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 671-698). Bethesda, MD: National Association of School Psychologists. If students have low general reading skills, RC strategy intervention should be time limited and the use of such strategies should be actively taught, expected, supported, and rewarded in Tier 1, general education and/or content area classes. Willingham, D.T. (2006/07). The usefulness of Brief instruction in reading comprehension strategies. American Educator, Winter, 39-50.

The Problem of “Fluency” Experienced--and Often Powerful--Teachers Push Back When R-CBM is (Inappropriately) Interpreted and Communicated as “Fluency” How Did This Happen? Errors in Judgment(s), Confusing Constructs with Behaviors Tested Tendency toward Reductionism

An Example of Reading Curriculum-Based Measurement (R-CBM) It was a pretty good composition. I felt proud knowing it was the best one at my school. After I’d read it five times, I was impatient to start reading it out loud. I followed the book’s directions again. First I read the composition out loud without trying to sound impressive, just to hear what the words sounded like. How many Words did Billy Read Correctly--Ask the audience. About 40--38 to be exact What is Billy’s accuracy? have the audience estimate. Actual accuracy is 73%, which is poor reading. Our judgments can get us in trouble. That’s why we use objective tests. Billy, 4th Grader

Oversimplifying the NRP Phonemic Awareness Alphabetic Understanding Fluency Vocabulary Comprehension ALL These Skills General Reading Skill Strands are NOT BOXES- Low Scores “in the Box” Led to TEACHING the Things in the Box SEPARATELY Despite some attempts to put a standard measure of oral reading “into a box” when a student reads aloud, we are assigned a student’s overall general reading achievement skills. The best way to know how well a S reads is to LISTEN to them...under standard conditions. Now we know that some of you will be anxious that you know students who can do this, but don’t comprehend. That’s a complex issue we need to understand more fully. Comprehension results FROM reading but is NOT “reading”...

Consequences of (Mis) Interpretation as Fluency Teacher Push Back and Shallow Intervention If R-CBM Interpreted As Fluency If Interpreted R-CBM(Correctly) As General Reading Ability Goal is to Read Fast Goal is to Read WELL Interventions Emphasize Speed Interventions Emphasize Quality Interventions are “Slices” or Bandaids Interventions are Integrated or Bandages

Research Evidence of “Fluency-Driven” Intervention or “Reading Faster”? Implementing repeated reading and wide reading interventions without more formative intervention is not likely to be valuable (p. 9) Wexler, J., Vaughn, S., Roberts, G., & Denton, C. A. (2010). The efficacy of repeated reading and wide reading practice for high school students with severe reading disabilities. Learning Disabilities Research & Practice, 25, 2-10. Our results indicate that repeated reading does not qualify as an evidence-based or promising practice for students with or at risk for learning disabilities (p. 276) Chard, D. J., Ketterlin-Geller, L. R., Baker, S. K., Doabler, C., & Apichatabutra, C. (2009). Repeated reading interventions for students with learning disabilities: Status of the evidence. Exceptional Children, 75, 263-281.

A Student with a Significant Reading Discrepancy Potentially Severe Educational Need

Poor Reading Quality as Well as Quantity Reads Accurately? Reads Efficiently with Automaticity? Reads with Expression (Prosody)? Effective Strategy for Unknown Words? Errors Distort or Preserve Meaning? Self Corrects Errors (Comprehension Self-Monitoring)? Adjusts Pace When Text Difficulty Changes?

A Poor High School Reader Grade 8 Material < 10th percentile at beginning of Grade 8 55

It’s Not an Empirical Issue--The Evidence is Compelling and Overwhelming Comprehension Measure Criterion Measure Correlation R-CBM SAT Word Study .80 SAT Comprehension 0.91 Question Answering .66 .82 Recall .58 0.70 Cloze .71 0.72 Fuchs, L.S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9, 20-28.

It’s Not an Empirical Issue--The Evidence is Compelling and Overwhelming .90 .88 .85 .71 .72 Shinn, M.R., Good, R.H., Knutson, N., Tilly, W.D., & Collins, V. (1992). Curriculum-Based reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21(3), 458-478.

But, I KNOW and HAVE SEEN Word Callers!!!!!!! The Research The idea of word callers has gained popularity despite a lack of evidence that applies “to an appreciable number of poor readers” (Stanovich, 1986, p. 372). Stanovich, K.E. (1986). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21, 360-406.

But, I KNOW and HAVE SEEN Word Callers!!!!!!! The Practice 31 of 75 Third Grade Teachers responded to a postcard put in their box asking them “if you teach a third-grade student who can read fluently, but has difficulty comprehending text.” 41%! Hamilton, C., & Shinn, M.R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240.

But, I KNOW and HAVE SEEN Word Callers!!!!!!! Teachers Identified PAIRS of Kids in Their Classrooms: Word Callers (WC) Similarly “Fluent” Peers Read Fluently, But Didn’t Comprehend Read Equally as Fluently (Well) As WC, But DID COMPREHEND Teachers Predicted BOTH Pair’s R-CBM, Maze, and Oral Comprehension Question Answering Skills--THEN Both Pairs WERE TESTED Hamilton, C., & Shinn, M.R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240.

The “Word Caller”

Teachers Were Partially Right! WC Peers WC Peers Almost 2 Standard Deviations Lower Almost 1.3 Standard Deviations Lower WCs “Comprehended” Less Well But WRC Was ALSO MUCH Lower

So Where Do This Leave Us... Reading Comprehension is Complex and We Tend to Oversimplify and Overgeneralize Reading is Necessary, But Not Sufficient for Understanding Reading Comprehension Comes From Reading, But is MORE Than Reading

“Comprehension” is Oversimplified A-Rod hit the cover off of the ball, but ended the game with a 6-4-3 double play. Inferential: Why would people from Beantown celebrate this? Why would this event mean different things in June than October? Factual: Who is A-Rod? What does “6” mean?

Another Common Example The procedure is actually quite simple. First, you arrange things into different groups. Of course, one pile may be sufficient depending on how much there is to do. If you have to go somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set. It is important not to overdo things. That is, it is better to do too few things at once than too many. In the short run, this may not seem important, but complications can easily arise. A mistake can be expensive as well. After the procedure is completed, one arranges the materials into different groups again. Then they can be put into their appropriate places. Eventually, they will be used once more and the whole cycle will have to be repeated. At first, the whole procedure will seem complicated. Soon, however, it will become just another fact of life. It is difficult to foresee any end the necessity for this task in the immediate future, but then one can never tell. 173-word paragraph with 14 sentences of words that are very short (about 4 letters) and rated as late Grade 6 by Flesch-Kincaid

Let Me Add Some Information How to Wash Clothes The procedure is actually quite simple. First, you arrange things into different groups. Of course, one pile may be sufficient depending on how much there is to do. If you have to go somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set. It is important not to overdo things. That is, it is better to do too few things at once than too many. In the short run, this may not seem important, but complications can easily arise. A mistake can be expensive as well. After the procedure is completed, one arranges the materials into different groups again. Then they can be put into their appropriate places. Eventually, they will be used once more and the whole cycle will have to be repeated. At first, the whole procedure will seem complicated. Soon, however, it will become just another fact of life. It is difficult to foresee any end the necessity for this task in the immediate future, but then one can never tell.

General Reading Skills The Bigger Deficits Here • Life Experience • Content Knowledge • Activation of Prior Knowledge • Knowledge about Texts Language • Oral Language Skills • Knowledge of Language Structures • Vocabulary • Cultural Influences And Here Reading Comprehension Oral Reading is the EASIEST to Measure--Let’s Get This Down and Add MORE Tools Knowledge Fluency* We Refer to It as General Reading Skills And the MOST Unmotivated Here For Some, the Hardest Thing They’ll Ever Do The Easiest Thing To Teach The barrier--the belief that oral reading for 1 minute should be discounted as a general progress monitoring tool because of I know a kid who reads fluently, but Comprehension depends on lots of things besides general reading ability Like what you are being asked to comprehend. How many in the audience can’t “comprehend” their college philosophy texts? or a stats text? Why? general reading abilty or something else? Let’s take comprehension apart using a graphic organizer of Torgeson’s we’ve modified slightly. And when we get good at oral reading, we’ll add Curriculum-Based EVALUATION skills to assess this other areas. The Longer It Takes... • Prosody • Automaticity/Rate • Accuracy • Decoding • Phonemic Awareness Metacognition • Motivation & Engagement • Active Reading Strategies • Monitoring Strategies • Fix-Up Strategies *modified slightly from presentations by Joe Torgesen, Ph.D. Co-Director, Florida Center for Reading Research; www.fcrr.org

Comes From Reading, But Is Not Reading Reading Early, Well, and Wide (EWW) is Critical for Reading Comprehension. See: Hunter, P.C. (2012). It's not complicated! What I know for sure about helping our students of color become successful readers. New York, NY: Scholastic. Cunningham, A.E., & Stanovich, K.E. (1998). What reading does for the mind. American Educator, 8-15. Denton, C.A., & Vaughn, S. (2010). Preventing and remediating reading difficulties: Perspectives from research. In M. R. Shinn & H. M. Walker (Eds.), Interventions for achievement and behavior problems in a three-tier model, including RTI (pp. 469-500). Bethesda, MD: National Association of School Psychologists. Linan-Thompson, S., & Vaughn, S. (2007). Research-based methods of reading instruction for English language learners Grades K-4. Alexandria, VA: Association for Supervision and Curriculum Development. http://www.youtube.com/watch?feature=player_embedded&v=RiP-ijdxqEc

Comes From Reading, But Is Not Reading Very Few Students Receive Evidence-Based or Sufficiently Explicit/Systematic Comprehension Strategy Instruction, Especially in Navigating Expository/Informational Text and Vocabulary, Learning the “Right Words” the “Right Way.” This is an INSTRUCTIONAL PROBLEM, Not An Assessment Problem! See: Feldman, K., & Kinsella, K. (2005). Narrowing the language gap: The case for explicit vocabulary instruction. In Scholastic (Ed.), Read About: . New York, NY Kamil, M.L., Borman, G.D., Dole, J., Kral, C.C., Salinger, T., & Torgesen, J. (2008). Improving Adolescent Literacy: Effective Classroom and Intervention Practices: A Practice Guide. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Educational Sciences, U.S. Department of Education. Marchand-Martella, N.E., Martella, R.C., Modderman, S.L., Petersen, H., & Pan, S. (2013). Key areas of effective adolescent literacy programs. Education and Treatment of Children, 36, 161-184. Torgesen, J., Houston, D.D., Rissman, L.M., Decker, S.M., Roberts, G., Vaughn, S., . . . Lesaux, N. (2007). Academic literacy instruction for adolescents: A guidance document from the Center on Instruction. Portsmouth, NH: RMC Research Corporation, Center on Instruction.

We Need to Expand Our Assessment Repertoire We Need to Add a Variety of Ways to Authentically Assess Comprehension in the Materials We Expect Students to be Able to Read and UNDERSTAND See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology IV (pp. 671-698). Bethesda, MD: National Association of School Psychologists. Howell, K.W., & Nolet, V. (1999). Curriculum-based evaluation: Teaching and decision making (3rd ed.). Atlanta, GA: Wadsworth Publishing Company.

Big Ideas When students read passages aloud for 1 minute, we are obtaining a holistic assessment of general reading skills. A strong empirical relation has been demonstrated between R-CBM and measures of general reading and reading comprehension (RC). Persuasive empirical data are not always persuasive of teacher opinion. General reading skills are necessary, but not sufficient for comprehension. RC is not a single “thing."  Judgments are based on: What a student is expected to read. How the student will be assessed. RC is influenced by a number of important variables, including General reading ability Language, especially vocabulary and familiarity with cultural idioms Knowledge, especially content knowledge and knowledge of the world Metacognitive strategies, including motivation and interest