Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Problem of Comprehension: A Potential Barrier to Reading Curriculum-Based Measurement as a Progress Monitoring and Screening Instrument Mark R. Shinn,

Similar presentations


Presentation on theme: "The Problem of Comprehension: A Potential Barrier to Reading Curriculum-Based Measurement as a Progress Monitoring and Screening Instrument Mark R. Shinn,"— Presentation transcript:

1

2 The Problem of Comprehension: A Potential Barrier to Reading Curriculum-Based Measurement as a Progress Monitoring and Screening Instrument Mark R. Shinn, Ph.D. Professor and School Psychology Program National Louis University, Skokie, IL February 21st, 2013

3 1 of 6 members of Technical Review Panel, National Center for Student Progress Monitoring, USDE/OSEP Editor and Contributor to 2 Major Texts on CBM Author of More than 75 Refereed Journal Articles and Book Chapters on the Topic of CBM, Progress Monitoring, and Screening My Area of Expertise

4 Mark R. Shinn, Ph.D. Serves as a Paid Consultant for Pearson Assessment for their AIMSweb product that provides CBM assessment materials and organizes and report the information from 3 tiers, including RTI. He provides technical support and training. Mark R. Shinn, Ph.D. Serves as a Consultant for Cambium/Voyager/Sopris for their Vmath product, a remedial mathematics intervention but has no financial interests. He helped them develop their progress monitoring system. Mark R. Shinn, Ph.D. Serves as a Consultant for McGraw-Hill Publishing for their Jamestown Reading Navigator (JRN) product and receives royalties.He helped them develop their progress monitoring system. Mark R. Shinn, Ph.D. Serves as a Member of the National Advisory Board for the CORE (Consortium on Reaching Excellence) and receives a stipend for participation. He provides training and product development advice. Disclosure

5 Learning to Read is Critical for School (and Life) Success

6 Frequent Formative Assessment Like Weekly Use of Reading Curriculum-Based Measurement (R-CBM) is Among our Most Powerful Tools! Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge. And the Number 1 Most Powerful TEACHING Variable

7 What about comprehension ? What about word callers ? I know this kid who... reads fluently but doesnt understand a THING they read... BUT...

8 Will Be Available in pdf format iBook format Presentation is Based on the Following White Paper in PRODUCTION A glossy and official Pearson version will be finished soon and sent to you. Shinn, M.R. (2013). The problem of comprehension: A potential barrier to Reading Curriculum-Based Measurement (R-CBM) as a progress monitoring and screening instrument. Minneapolis, MN: Pearson Assessment.

9 References on CBM Criterion Related Validity Deno, S. L., Marston, D., Shinn, M. R., & Tindal, G. (1983). Oral reading fluency: A simple datum for scaling reading disability. Topics in Learning and Learning Disability, 2, Fuchs, L. S., Fuchs, D., Hosp, M. K., & Jenkins, J. R. (2001). Oral reading fluency as an indicator of reading competence: A theoretical, empirical, and historical analysis. Scientific Studies of Reading, 5, Fuchs, L. S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9, Hamilton, C., & Shinn, M. R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32, Miura Wayman, M., Wallace, T., Ives Wiley, H., Ticha, R., & Espin, C. (2007). Literature synthesis on curriculum-based measurement in reading. The Journal of Special Education, 41, Shinn, M. R., Good, R. H., Knutson, N., Tilly, W. D., & Collins, V. (1992). Curriculum-Based reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21,

10 10 Accessing Reading Materials markshinn.org 1. Click on the Downloads for Professionals Icon 2. Click on the Presentations and Handouts Folder 3.Click on AIMSweb Webinar R-CBM and the Problem of Comprehension (Sponsored by Pearson) 2013 FolderClick on AIMSweb Webinar R-CBM and the Problem of Comprehension (Sponsored by Pearson) 2013 Folder 10

11 Big Ideas 1.When students read passages aloud for 1 minute, we are obtaining a holistic assessment of general reading skills. 2.A strong empirical relation has been demonstrated between R-CBM and measures of general reading and reading comprehension (RC). 3.Persuasive empirical data are not always persuasive of teacher opinion. 4.General reading skills are necessary, but not sufficient for comprehension. 5.RC is not a single thing." Judgments are based on: (a)What a student is expected to read. (b)How the student will be assessed. 6.RC is influenced by a number of important variables, including (a)General reading ability (b)Language, especially vocabulary and familiarity with cultural idioms (c)Knowledge, especially content knowledge and knowledge of the world (d)Metacognitive strategies, including motivation and interest

12 2 Other Big Ideas We Cant Discuss Today 1.If educators are concerned about specific students RC, then we must have a systematic assessment process to address these concerns, starting with ruling out general reading skill deficits, but also including interviews, observations, and diagnostic assessment. See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp ). Bethesda, MD: National Association of School Psychologists. If students have low general reading skills, RC strategy intervention should be time limited and the use of such strategies should be actively taught, expected, supported, and rewarded in Tier 1, general education and/or content area classes. See: Willingham, D.T. (2006/07). The usefulness of Brief instruction in reading comprehension strategies. American Educator, Winter,

13 2 Other Big Ideas We Cant Discuss Today 1.If educators are concerned about specific students RC, then we must have a systematic assessment process to address these concerns, starting with ruling out general reading skill deficits, but also including interviews, observations, and diagnostic assessment. See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp ). Bethesda, MD: National Association of School Psychologists. If students have low general reading skills, RC strategy intervention should be time limited and the use of such strategies should be actively taught, expected, supported, and rewarded in Tier 1, general education and/or content area classes. Willingham, D.T. (2006/07). The usefulness of Brief instruction in reading comprehension strategies. American Educator, Winter,

14 The Problem of Fluency Experienced--and Often Powerful--Teachers Push Back When R-CBM is (Inappropriately) Interpreted and Communicated as Fluency How Did This Happen? –Errors in Judgment(s), Confusing Constructs with Behaviors Tested –Tendency toward Reductionism

15 An Example of Reading Curriculum-Based Measurement (R-CBM) It was a pretty good composition. I felt proud knowing it was the best one at my school. After Id read it five times, I was impatient to start reading it out loud. I followed the books directions again. First I read the composition out loud without trying to sound impressive, just to hear what the words sounded like. Billy, 4th Grader

16 Phonemic Awareness Alphabetic Understanding Fluency Vocabulary Comprehension Oversimplifying the NRP Strands are NOT BOXES- Low Scores in the Box Led to TEACHING the Things in the Box SEPARATELY ALL These Skills General Reading Skill

17 Consequences of (Mis) Interpretation as Fluency Teacher Push Back and Shallow Intervention If R-CBM Interpreted As Fluency If Interpreted R-CBM(Correctly) As General Reading Ability Goal is to Read FastGoal is to Read WELL Interventions Emphasize SpeedInterventions Emphasize Quality Interventions are Slices or Bandaids Interventions are Integrated or Bandages

18 Research Evidence of Fluency- Driven Intervention or Reading Faster? Our results indicate that repeated reading does not qualify as an evidence-based or promising practice for students with or at risk for learning disabilities (p. 276) Chard, D. J., Ketterlin-Geller, L. R., Baker, S. K., Doabler, C., & Apichatabutra, C. (2009). Repeated reading interventions for students with learning disabilities: Status of the evidence. Exceptional Children, 75, Implementing repeated reading and wide reading interventions without more formative intervention is not likely to be valuable (p. 9) Wexler, J., Vaughn, S., Roberts, G., & Denton, C. A. (2010). The efficacy of repeated reading and wide reading practice for high school students with severe reading disabilities. Learning Disabilities Research & Practice, 25, 2-10.

19 Potentially Severe Educational Need A Student with a Significant Reading Discrepancy

20 Poor Reading Quality as Well as Quantity Reads Accurately? Reads Efficiently with Automaticity? Reads with Expression (Prosody)? Effective Strategy for Unknown Words? Errors Distort or Preserve Meaning? Self Corrects Errors (Comprehension Self-Monitoring)? Adjusts Pace When Text Difficulty Changes?

21 A Poor High School Reader 55 Grade 8 Material < 10th percentile at beginning of Grade 8

22 Its Not an Empirical Issue--The Evidence is Compelling and Overwhelming Comprehension MeasureCriterion MeasureCorrelation R-CBMSAT Word Study.80 SAT Comprehension0.91 Question AnsweringSAT Word Study.66 SAT Comprehension.82 RecallSAT Word Study.58 SAT Comprehension0.70 ClozeSAT Word Study.71 SAT Comprehension0.72 Fuchs, L.S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9,

23 Its Not an Empirical Issue--The Evidence is Compelling and Overwhelming Shinn, M.R., Good, R.H., Knutson, N., Tilly, W.D., & Collins, V. (1992). Curriculum-Based reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21(3),

24 But, I KNOW and HAVE SEEN Word Callers!!!!!!! The Research The idea of word callers has gained popularity despite a lack of evidence that applies to an appreciable number of poor readers (Stanovich, 1986, p. 372). Stanovich, K.E. ( 1986 ). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21,

25 But, I KNOW and HAVE SEEN Word Callers!!!!!!! The Practice 31 of 75 Third Grade Teachers responded to a postcard put in their box asking them if you teach a third-grade student who can read fluently, but has difficulty comprehending text. 41%! Hamilton, C., & Shinn, M.R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32,

26 But, I KNOW and HAVE SEEN Word Callers!!!!!!! Teachers Identified PAIRS of Kids in Their Classrooms: Hamilton, C., & Shinn, M.R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32, Word Callers (WC)Similarly Fluent Peers Read Fluently, But Didnt Comprehend Read Equally as Fluently (Well) As WC, But DID COMPREHEND Teachers Predicted BOTH Pairs R-CBM, Maze, and Oral Comprehension Question Answering Skills--THEN Both Pairs WERE TESTED

27 The Word Caller

28 Teachers Were Partially Right! WCs Comprehended Less Well WC Peers But WRC Was ALSO MUCH Lower WC Peers Almost 2 Standard Deviations Lower Almost 1.3 Standard Deviations Lower

29 So Where Do This Leave Us... 1.Reading Comprehension is Complex and We Tend to Oversimplify and Overgeneralize Reading is Necessary, But Not Sufficient for Understanding Reading Comprehension Comes From Reading, But is MORE Than Reading

30 Comprehension is Oversimplified A-Rod hit the cover off of the ball, but ended the game with a double play. Factual: Who is A-Rod? What does 6 mean? Inferential: Why would people from Beantown celebrate this? Why would this event mean different things in June than October?

31 Another Common Example 173-word paragraph with 14 sentences of words that are very short (about 4 letters) and rated as late Grade 6 by Flesch- Kincaid The procedure is actually quite simple. First, you arrange things into different groups. Of course, one pile may be sufficient depending on how much there is to do. If you have to go somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set. It is important not to overdo things. That is, it is better to do too few things at once than too many. In the short run, this may not seem important, but complications can easily arise. A mistake can be expensive as well. After the procedure is completed, one arranges the materials into different groups again. Then they can be put into their appropriate places. Eventually, they will be used once more and the whole cycle will have to be repeated. At first, the whole procedure will seem complicated. Soon, however, it will become just another fact of life. It is difficult to foresee any end the necessity for this task in the immediate future, but then one can never tell.

32 Let Me Add Some Information The procedure is actually quite simple. First, you arrange things into different groups. Of course, one pile may be sufficient depending on how much there is to do. If you have to go somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set. It is important not to overdo things. That is, it is better to do too few things at once than too many. In the short run, this may not seem important, but complications can easily arise. A mistake can be expensive as well. After the procedure is completed, one arranges the materials into different groups again. Then they can be put into their appropriate places. Eventually, they will be used once more and the whole cycle will have to be repeated. At first, the whole procedure will seem complicated. Soon, however, it will become just another fact of life. It is difficult to foresee any end the necessity for this task in the immediate future, but then one can never tell. How to Wash Clothes

33 ReadingComprehension Knowledge Fluency* We Refer to It as General Reading Skills Metacognition Languag e Prosody Prosody Automaticity/Rate Automaticity/Rate Accuracy Accuracy Decoding Decoding Phonemic Awareness Phonemic Awareness Oral Language Skills Oral Language Skills Knowledge of Language Knowledge of Language Structures Structures Vocabulary Vocabulary Cultural Influences Cultural Influences Life Experience Life Experience Content Knowledge Content Knowledge Activation of Prior Activation of Prior Knowledge Knowledge Knowledge about Knowledge about Texts Texts Motivation & Motivation & Engagement Engagement Active Reading Active Reading Strategies Strategies Monitoring Strategies Monitoring Strategies Fix-Up Strategies Fix-Up Strategies *modified slightly from presentations by Joe Torgesen, Ph.D. Co- Director, Florida Center for Reading Research; For Some, the Hardest Thing Theyll Ever Do The Easiest Thing To Teach The Longer It Takes... The Bigger Deficits Here And Here And the MOST Unmotivated Here Oral Reading is the EASIEST to Measure-- Lets Get This Down and Add MORE Tools

34 Comes From Reading, But Is Not Reading Reading Early, Well, and Wide (EWW) is Critical for Reading Comprehension. See: Hunter, P.C. (2012). It's not complicated! What I know for sure about helping our students of color become successful readers. New York, NY: Scholastic. Cunningham, A.E., & Stanovich, K.E. (1998). What reading does for the mind. American Educator, Denton, C.A., & Vaughn, S. (2010). Preventing and remediating reading difficulties: Perspectives from research. In M. R. Shinn & H. M. Walker (Eds.), Interventions for achievement and behavior problems in a three-tier model, including RTI (pp ). Bethesda, MD: National Association of School Psychologists. Linan-Thompson, S., & Vaughn, S. (2007). Research-based methods of reading instruction for English language learners Grades K-4. Alexandria, VA: Association for Supervision and Curriculum Development.

35 Comes From Reading, But Is Not Reading Very Few Students Receive Evidence-Based or Sufficiently Explicit/Systematic Comprehension Strategy Instruction, Especially in Navigating Expository/Informational Text and Vocabulary, Learning the Right Words the Right Way. This is an INSTRUCTIONAL PROBLEM, Not An Assessment Problem! See: Feldman, K., & Kinsella, K. (2005). Narrowing the language gap: The case for explicit vocabulary instruction. In Scholastic (Ed.), Read About:. New York, NY Kamil, M.L., Borman, G.D., Dole, J., Kral, C.C., Salinger, T., & Torgesen, J. (2008). Improving Adolescent Literacy: Effective Classroom and Intervention Practices: A Practice Guide. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Educational Sciences, U.S. Department of Education. Marchand-Martella, N.E., Martella, R.C., Modderman, S.L., Petersen, H., & Pan, S. (2013). Key areas of effective adolescent literacy programs. Education and Treatment of Children, 36, Torgesen, J., Houston, D.D., Rissman, L.M., Decker, S.M., Roberts, G., Vaughn, S.,... Lesaux, N. (2007). Academic literacy instruction for adolescents: A guidance document from the Center on Instruction. Portsmouth, NH: RMC Research Corporation, Center on Instruction.

36 We Need to Expand Our Assessment Repertoire We Need to Add a Variety of Ways to Authentically Assess Comprehension in the Materials We Expect Students to be Able to Read and UNDERSTAND See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology IV (pp ). Bethesda, MD: National Association of School Psychologists. Howell, K.W., & Nolet, V. (1999). Curriculum-based evaluation: Teaching and decision making (3rd ed.). Atlanta, GA: Wadsworth Publishing Company.

37 Big Ideas 1.When students read passages aloud for 1 minute, we are obtaining a holistic assessment of general reading skills. 2.A strong empirical relation has been demonstrated between R-CBM and measures of general reading and reading comprehension (RC). 3.Persuasive empirical data are not always persuasive of teacher opinion. 4.General reading skills are necessary, but not sufficient for comprehension. 5.RC is not a single thing." Judgments are based on: (a)What a student is expected to read. (b)How the student will be assessed. 6.RC is influenced by a number of important variables, including (a)General reading ability (b)Language, especially vocabulary and familiarity with cultural idioms (c)Knowledge, especially content knowledge and knowledge of the world (d)Metacognitive strategies, including motivation and interest


Download ppt "The Problem of Comprehension: A Potential Barrier to Reading Curriculum-Based Measurement as a Progress Monitoring and Screening Instrument Mark R. Shinn,"

Similar presentations


Ads by Google