2. Schoolwide Reading Day 1: 4.0 Curriculum Based Measurement AIMSweb Reading CBM (R-CBM) and Maze Cohort 7 Middle Schools

3. Acknowledgements Content in this presentation is based upon the work of –AIMSweb and DIBELS developers –Beth Harn, University of Oregon –MiBLSi leadership and trainers Gloria Johnson Margie McGlinchey Jim Weaver

4. OVERVIEW OF CURRICULUM-BASED MEASUREMENT AS A GENERAL OUTCOME MEASURE Mark R. Shinn, Ph.D. Michelle M. Shinn, Ph.D. Lisa A. Langell, M.A., S.Psy.S.

5. CBM is a General Outcome Measure Used for Scientific Reasons Based on Evidence  Reliable and valid indicator of student achievement  Simple, efficient, and of short duration to facilitate frequent administration by teachers  Provides assessment information that helps teachers plan better instruction  Sensitive to the improvement of students’ achievement over time  Easily understood by teachers and parents  Improves achievement when used to monitor progress

6. Things to Always Remember About CBM  Designed to serve as “indicators” of general reading achievement: CBM probes don’t measure everything, but measure the important things.  Standardized tests to be given, scored, and interpreted in a standard way  Researched with respect to psychometric properties to ensure accurate measures of learning

7. Big Ideas of Benchmark (Tier 1) Assessment Benchmarking allows us to add systematic Formative Evaluation to current practice. For Teachers (and Students) Early Identification of At Risk Students Instructional Planning Progress Monitoring For Parents Opportunities for Communication/Involvement Accountability For Administrators Resource Allocation/Planning and Support Accountability

8. Benchmark (Tier 1): 3x per year Strategic Monitoring (Tier 2): 1x per month for select students at risk for educational difficulties (Optional) Progress Monitor (Tier 3): Intensive assessment with adjustable frequency that matches need

9. Reading Assessment Four Kinds or Purposes Outcome - Assessments that provide a bottom-line evaluation of the effectiveness of the reading program. Screening - Assessments that are administered to determine which children are at risk for reading difficulty and who will need additional intervention. Diagnosis - Assessments that help teachers plan instruction by providing in-depth information about students’ skills and instructional needs. Progress Monitoring - Assessments that determine if students are making adequate progress or need more intervention to achieve grade level reading outcomes. An effective, comprehensive, reading program includes reading assessments to accomplish four purposes: Source: Reading First Initiative: Secretary’s Leadership Academy

10. Evaluating status of reading assessments in school Part 1 1.Describe the reading assessments your school uses to evaluate student reading outcomes/achievement. 2.Use the first section of the worksheet. Team Time

11. Evaluating status of reading assessments in school Part 2 1.Describe the reading assessments your school uses to determine which children are at risk for reading difficulty and who will need additional intervention. 2.Use the second section of the worksheet. Team Time

12. Evaluating status of reading assessments in school Part 3 1.Describe the reading assessments your school uses to help teachers plan instruction by providing in-depth information about students’ skills and instructional needs. 2.Use the third section of the worksheet. Team Time

13. Evaluating status of reading assessments in school Part 4 1.Describe the reading assessments your school uses to determine if students are making adequate progress or need more intervention to achieve grade level reading outcomes. 2.Use the fourth section of the worksheet. Team Time

14. General Outcome Measurements from Other Fields Medicine measures height, weight, temperature, and/or blood pressure. Department of Labor measures the Consumer Price Index. Wall Street measures the Dow-Jones Industrial Average. Companies report earnings per share. McDonald’s measures how many hamburgers they sell.

15. Common Characteristics: General Outcome Measures (GOMS) Simple, accurate, and reasonably inexpensive in terms of time and materials. Considered so important to doing business well that they are routine. Are collected on an ongoing and frequent basis. Shape/inform a variety of important decisions.

16. Origins of CBM as General Outcome Measures Curriculum-Based Measurement (CBM) was developed more than 30 years ago by Stanley Deno at the University of Minnesota through a federal contract to develop a reliable and valid measurement system for evaluating basic skills growth. CBM is supported by more than 30 years of school-based research by the US Department of Education. Supporting documentation can be found in more than 150 articles, book chapters, and books in the professional literature describing the use of CBM to make a variety of important educational decisions.

17. CBM is Designed to Provide Educators With: The same kind of evaluation technology as other professions! Powerful measures that are: Simple Accurate Efficient indicators of student achievement that guide/inform a variety of decisions

18. What Works? See Kavale (2005), Learning Disabilities, 13, 127-138 and other sources TreatmentEffect Size Applied Behavior Analysis+ 1.00 CBM+Graphing+Formative Evaluation + reinforcement + 1.00 Explicit Instruction and Problem Solving +.70 to 1.50 Comprehension Strategies+ 1.00 Math Interventions+.60 to 1.10 Writing Interventions+.50 to.85

19. Reading - Curriculum Based Measurement (R-CBM) Students read aloud meaningful, connected text from AIMSweb passages for 1 minute Number of words read correct and number of errors are counted and Reported as WRC/errors Area AssessedTimingArrangementWhat is Scored CBM Oral Reading Fluency (R-CBM) 1 minuteIndividual# of Words Read Correct (WRC) and the # of Errors

20. Reading - Curriculum Based Measurement (R-CBM)

21. Observation Questions What did you observe about this child’s reading? What kind of mistakes did you hear? Is she a good reader?

22. R-CBM Measures Are designed to serve as “indicators” of general reading achievement. R-CBM doesn’t measure everything, but measures the important things. Are Standardized tests to be given, scored, and interpreted in a standard way. Are researched with respect to psychometric properties to ensure accurate measures of learning.

23. R-CBM Measures (Continued) Are Sensitive to improvement in Short Periods of time. Also tells us how students earned their scores (Qualitative Information). Designed to be as short as possible to ensure its “do ability.” Are linked to decision making for promoting positive achievement and Problem- Solving.

24. Administration and Scoring AIMSweb Reading Curriculum Based Measurement (R-CBM)

25. Administration and Scoring of R-CBM What Examiners Need to Do... Before testing students While testing students After testing students

26. Before: Things You Need Before Testing Standard Reading Assessment Passage Student Copy: No numbers Between 250-300 words (exception: 1st grade) An informative first sentence Same font style and size Text without pictures

27. Before: Things You Need Before Testing (Continued) Standard Reading Assessment Passage Examiner Copy: Pre-numbered so they can be scored quickly and immediately.

28. A Note About - AIMSweb’s Standard Reading Passages Are written to represent general curriculum or be “curriculum independent.” Allow decision making about reading growth, regardless of between-school, between- school district, between-teacher differences in reading curriculum. Are graded to be of equal difficulty. Have numerous alternate forms for testing over time without practice effects.

29. Before: Also Needed – Additional Assessment Aids A List of Students to be Tested Stop Watch (required) Clipboard Pencil

30. A Note About Timers Make friends with your timer : ) –Start/stop –Clear –Set to 1 minute, count down –Practice 3 second internal count No wall clocks or kitchen timers please!

31. Before: Setting up Assessment Environment Flexible assessment environments could include… A set-aside place in the classroom A reading station in the hallway Reading stations in the media center, cafeteria, gym, or empty classrooms

32. While: Things You Need to Do While Testing Follow the standardized directions: R-CBM is a standardized test Read directions verbatim and score by the rules every time for every student Remember it’s about testing, not teaching Don’t teach or correct errors Don’t practice reading the passages Remember Best, not Fastest reading Sit across from, not beside, the student

33. While: R-CBM Standard Directions for 1 Minute Administration 1.Place the unnumbered copy in front of the student. numbered 2.Place the numbered copy in front of you, but shielded so the student cannot see what you record. 3.Say: When I say ‘Begin,’ start reading aloud at the top of this page. Read across the page (DEMONSTRATE BY POINTING). Try to read each word. If you come to a word you don’t know, I will tell it to you. Be sure to do your best reading. Are there any questions? (PAUSE) “” 4.Say “Begin” and start your stopwatch when the student says the first word. If the student fails to say the first word of the passage after 3 seconds, tell them the word, mark it as incorrect, then start your stopwatch. 5.Follow along on your copy. Put a slash ( / ) through words read incorrectly “.” 6.At the end of 1 minute, place a bracket ( ] ) after the last word and say, “Stop.” 7. Score and summarize by writing WRC/Errors

34. While: “Familiar” Shortened Directions When students are assessed frequently and know the directions. Say: When I say ‘Begin,’ start reading aloud at the top of this page.

35. While: Apply Standardized Scoring Rules ~Reviewing Scoring Rules in Training Workbook ~ Tab the scoring rules for easy reference Partner Work

36. What is a Word Read Correctly? Correctly pronounced words within context Words Self-Corrected within 3 seconds

37. What is an Error? Mispronunciation of the Word or Substitutions Omissions Stops or struggles with a letter for more than 3 seconds (examiner provides correct word)

38. What Is Not Incorrect? Not Counted as an Error (Neither a WRC or an Error) Repetitions Dialect Differences Insertions (Consider them Qualitative Errors)

39. While: Remember Emphasize Words Read Correctly (WRC) Follow Standardized Administration and Scoring to Get an Accurate Count 3-Second Rule No other corrections Discontinue rule Be polite Best, not fastest Interruptions

40. Things to Do After Testing Determine score immediately to ensure accurate results! Record score on front of booklet If doing multiple samples, organize your impressions of qualitative features

41. Calculating and Reporting Scores Count Total number of Words Read (TWR) Count the number of Errors (E) and subtract Report in standard format of WRC/Errors (72/3) TWR - E = WRC WRC/Errors

42. Example of Calculating Scores Juan finished reading after 1 minute at the 145th word, so he read 145 words total Juan also made 3 errors Therefore, his WRC was 142 with 3 errors Reported as 142/3

43. Determining the Median Score Administer 3 passages and take the median score List WRC scores, cross off the high score, cross off the low score Report the remaining, median or middle score

44. Determining the Median Score Examples 58/599/3

45. Practice Whole Group and Triads

46. Firewalkers/Gorilla! Group Practice

47. Gorilla! 99 TWR -6 E 93 WRC Score = 93/6

48. Firewalkers and Gorilla Oral Reading Fluency Scoring and Practice

49. Oral Reading Fluency Practice Breakout Activity Work in groups of 3Work in groups of 3 Materials neededMaterials needed –Clipboard (use one at school, do the best you can today without one) –Timer –Practice pages from notebook Team Time

50. Oral Reading Fluency Practice Activity Instructions for Completing the Activity: 1.Form a three-person group. You will take turns administering R-CBM to a lower performing Middle School student 2. There will be 3 rounds to the activity where your role will change with each round. ExaminerStudentObserver Round 1 MegCharles Wallace Calvin Round 2 Charles Wallace CalvinMeg Round 3 CalvinMegCharles Wallace

52. Examiner Follow the script on the page titled “Examiner.” Administer R-CBM to the “student.” Practice timing and scoring.

53. Student Follow the script on the page titled, “Student.” You will pretend to be a student and read from this script

54. Observer Use the R-CBM Assessment Integrity checklist on the page titled “Observer.” Observe the examiner and provide feedback.

55. After completing each round, discuss as a group how the administration went. Answer the following questions for the round in which you are the examiner: - What was easy? - What was difficult? - What do you need to practice? -How were the skills of the student you assessed? After completing each round…

56. Firewalkers and Gorilla Oral Reading Fluency Scoring, Extra Practice

57. Assessment Tools Qualitative Features Checklist Accuracy of Implementation Rating Scale (AIRS)

58. Qualitative Features Worth Noting

59. Accuracy of Implementation (AIRS)

60. Determining Inter-Rater Agreement Agreements/(Agreements + Disagreements) x 100 Example: Examiner 1 = 100 WRC Examiner 2 = 98 WRC Inter-rater agreement = 98%

61. Computation They agreed that the student read 98 of the words correct. They disagreed on 2 words correct. Agreements / (Agreements + Disagreements) x 100 98/(98 + 2)= 98/100 =.98.98 x 100 = 98%

62. Summary You now have the building blocks to begin AIMSweb Reading - Curriculum Based Measurement to ensure reading growth. –Practice to Automaticity --You’ll get more efficient –Get Checked Out with AIRS for accuracy/efficiency by a colleague –Stay in Tune by periodically checking AIRS

64. Huron County Grade Level Reading (Weaver & Johnson, 2006)

65. Administration and Scoring AIMSweb Mazes

66. Administration and Scoring of Mazes What Examiners Need to Do... Before testing studentsBefore testing students While testing studentsWhile testing students After testing studentsAfter testing students

67. Curriculum Based Measurement Reading Maze AreaTimingTest Arrangements What is Scored? CBM Maze Reading 3 minutes Individual, Small Group, or Classroom Group # of Correct Answers CBM Maze is designed to provide educators a more complete picture of students’ reading skills, especially when comprehension problems are suspected.

68. Curriculum Based Measurement Reading Maze (Continued) Maze is a multiple-choice cloze task that students complete while reading silently Students are presented with 150-400 word passages First sentence is left intact After the first sentence, every 7 th word is replaced with three word choices inside a parenthesis The three choices consist of 1) Near Distracter 2) Exact Match 3) Far Distracter

69. An example of CBM Maze

70. Observation Questions 1.What did you observe about Emma’s and Abby’s Maze performance? 2.What other conclusions can you draw?

71. CBM Maze is Used for Scientific Reasons Based on Evidence It is a reliable and valid indicator of student achievement. It is simple, efficient, and of short duration to facilitate frequent administration by teachers. It provides assessment information that helps teachers plan better instruction. It is sensitive to the improvement of students’ achievement over time. It is easily understood by teachers and parents. Improves achievement when used to monitor progress.

72. CBM Maze Measures Are designed to serve as “indicators” of general reading achievement. Are Standardized tests to be given, scored, and interpreted in a standard way. Are researched with respect to psychometric properties to ensure accurate measures of learning.

73. CBM Maze Measures (Continued) Are Sensitive to improvement in Short Periods of time. Designed to be as short as possible to ensure its “do ability.” Are linked to decision making for promoting positive achievement and Problem- Solving.

74. Administration and Scoring of CBM Maze What Examiners Need To Do... Before testing students While testing students After testing students

75. Before: Students Materials Needed What the Students Need for Testing: CBM Maze practice test (optional) Appropriate CBM Maze passages Pencils

76. Before: Examiner Materials Needed What the Tester Uses for Testing: Stopwatch Appropriate CBM Maze Answer Key Appropriate Standardized Directions

77. Before: Additional Assessment Aids A List of Students to be Tested Stop Watch (required)

78. A Note About Setting up Assessment Environment The classroom if assessing the entire class. A cluster of desks or small tables in the classroom for small group assessment. Individual desks or “stations” for individual assessment. Flexible assessment environments could include…

79. While: Things You Need To Do While Testing Attach a cover sheet that includes the practice test so that students do not begin the test right away. Do a simple practice test with younger students. Monitor to ensure students are circling answers instead of writing them. Be prepared to “Prorate” for students who may finish early. Try to avoid answering student questions. Adhere to the End of Timing. Follow the Standardized Directions

80. CBM Maze Standard Directions 1.Pass Maze tasks out to students. Have students write their names on the cover sheet, so they do not start early. Make sure they do not turn the page until you tell them to. 2.Say this to the student (s): When I say ‘Begin’ I want you to silently read a story. You will have 3 minutes to read the story and complete the task. Listen carefully to the directions. Some of the words in the story are replaced with a group of 3 words. Your job is to circle the 1 word that makes the most sense in the story. Only 1 word is correct. 3.Decide if a practice test is needed. Say... Let’s practice one together. Look at your first page. Read the first sentence silently while I read it out loud: ‘The dog, apple, broke, ran after the cat.’ The three choices are apple, broke, ran. ‘The dog apple after the cat.’ That sentence does not make sense. ‘The dog broke after the cat.’ That sentence does not make sense. ‘The dog ran after the cat.’ That sentence does make sense, so circle the word ran. (Make sure the students circle the word ran.)

81. CBM Maze Standard Directions ( Continued ) Let’s go to the next sentence. Read it silently while I read it out loud. ‘The cat ran fast, green, for up the hill. The three choices are fast, green, for up the hill. Which word is the correct word for the sentence? (The students answer fast) Yes, ‘The cat ran fast up the hill’ is correct, so circle the correct word fast. (Make sure students circle fast) Silently read the next sentence and raise your hand when you think you know the answer. (Make sure students know the correct word. Read the sentence with the correct answer) That’s right. ‘The dog barked at the cat’ is correct. Now what do you do when you choose the correct word? (Students answer ‘Circle it’. Make sure the students understand the task) That’s correct, you circle it. I think you’re ready to work on a story on your own.

82. CBM Maze Standard Directions (Continued) 1.Start the testing by saying... When I say ‘Begin’ turn to the first story and start reading silently. When you come to a group of three words, circle the 1 word that makes the most sense. Work as quickly as you can without making mistakes. If you finish a/ the page/first side, turn the page and keep working until I say ‘Stop’ or you are all done. Do you have any questions? 2.Then say, ‘Begin.’ Start your stopwatch. 3.Monitor students to make sure they understand that they are to circle only 1 word. 4.If a student finished before the time limit, collect the student’s Maze task and record the time on the student’s test booklet. 5.At the end of 3 minutes say: Stop. Put your pencils down. Please close your booklet. 6.Collect the Maze tasks.

83. CBM Maze Familiar Directions 1.After the students have put their names on the cover sheet, start the testing by saying... When I say ‘Begin’ turn to the first story and start reading silently. When you come to a group of three words, circle the 1 word that makes the most sense. Work as quickly as you can without making mistakes. If you finish a/ the page/first side, turn the page and keep working until I say ‘Stop’ or you are all done. Do you have any questions? 2.Then say, ‘Begin.’ Start your stopwatch. 3.Monitor students to make sure they understand that they are to circle only 1 word. 4.If a student finished before the time limit, collect the student’s Maze task and record the time on the student’s test booklet. 5.At the end of 3 minutes say: Stop. Put your pencils down. Please close your booklet. 6.Collect the Maze tasks.

84. Things To Do After Testing 1.Use the answer key and put a slash (/) through incorrect responses 2.Determine the Number of Responses Correct.

85. CBM Maze Scoring What is correct? The student’s circled the word that matches the correct word on the scoring template. What is incorrect? An answer is considered an error if the student: 1.Circles an incorrect word 2.Does not circle any of the 3 choices 3.Circles more than one of the 3 choices

86. A Note About Efficient Scoring 1.Compare the student’s answers to the correct answers on the scoring template. Mark a slash [/] through incorrect responses. 2.Count the total number of items up to the last item attempted. 3.Subtract the number of incorrect answers from the last item attempted. 4.Record the total number of correct answers on the cover sheet followed by the total number of errors (e.g., 35/2).

87. CBM Maze Prorating 1.When the student finished must be recorded and the number of correct answers counted. For example, the student may have finished in 2 minutes and correctly answered 40 items. 2.Convert the time taken in seconds. (2 minutes = 120 sec) 3.Divide the number of seconds by the number correct. (120/40 = 3) 4.Calculate the number of seconds in the full 3 minutes. (3 minutes = 180 seconds) 5.Divide the number of full seconds by the calculated value from step 3. (180/3 = 60). This student’s score would be 60. If a student finishes all the items before 3 minutes, the score can be prorated.

88. Summary You now have the building blocks to begin CBM Maze to ensure literacy growth. Practice to Automaticity --You’ll get more efficient Get Checked Out with AIRS for accuracy/efficiency by a colleague Stay in Tune by periodically checking with AIRS

89. Practice Exercise 1: Let’s Score

90. Practice Exercise 2: Let’s Score

91. Assessment Logistics Pulling It Together Benchmark Assessment

92. Logistics of Data Collection Plan and schedule data collection. Organize resources. Collect the data. Enter the data. Use data for educational decision making.

93. Data Collection Benchmark Assessment: –Assess all children three times/year (e.g., fall, winter, spring). Progress Monitoring: –Assess children needing strategic support/monitoring more frequently (e.g., monthly). –Assess students needing more intensive intervention weekly.

94. Planning for Benchmark Assessment Decisions –When will the data be collected? –Who will collect the data? –How will the data be collected? What data collection approach will be used?

95. Who Will Collect Benchmark Data? Options: –Classroom teachers and assistants –Specialists and support staff –Principals and administrators –Trained volunteers –Practicum students Considerations: –Resources Number and availability of staff Interest Budgetary resources Training needs –Timeline –Approach

96. Test All Students All students participate in AIMSweb including special education students

97. What Data Collection Approach? Options: –Within classroom –School-wide: one day –School-wide: multiple days

98. Within Classroom Approach Who collects the data?Classroom teachers and assistants Where?In the classroom How?Teachers set aside time (e.g., 30 minutes a day for 4 days) to assess each child in the room AdvantagesDisadvantages Teachers test their own students Less disruptive to school in general Detracts from instructional time Requires more days May be more difficult to organize and use data system-wide Does not improve communication and encourage team approach to decision making Adapted from B. Harn (2000)

99. One-day School-wide Approach Who collects the data?Data collection team: support staff, trained volunteers, specialists, educational assistants, teachers (1 data collector for 20-25 students) Where?A large, central location with many tables and places for students to be assessed and to wait their turn (e.g., library, multipurpose room, cafeteria). How?A schedule is set for classrooms to come to a central location where the team assess all day. AdvantagesDisadvantages Minimal classroom disruption (e.g., 30 minutes per class) All data collected in one day School-wide effort improves communication and enhances effort to organize and use data system-wide. Need a large team Scheduling and logistics issues (e.g., need location for assessments; services may be disrupted for all students (e.g., computer lab, library). If using support staff, specialized services for students may be disrupted Teachers not involved in assessing own students Adapted from B. Harn (2000)

100. Multi-day School-wide Approach Who collects the data?Data collection team: support staff, trained volunteers, specialists, educational assistants, teachers Where?Central location or temporary area in/near the classroom How?The team either goes to the classroom and tests students in/near each classroom or classrooms go to team location AdvantagesDisadvantages Requires smaller core team Teachers may be more involved in data collection process Less disruption to school in general No need for central location School-wide effort improves communication and enhances effort to organize and use data system-wide. Takes longer to collect data on all students Testing team may need to move from one location to another If using support staff, specialized services for students may be disrupted Adapted from B. Harn (2000)

101. Considerations Goal: To collect valid and reliable benchmark data as efficiently and economically as possible, with minimal disruption Before deciding on an approach to use, consider: –Number of students to be assessed –School calendar and events –Timeline for completion of assessment –Availability of resources

102. Benchmark Assessment Time Middle School FallWinterSpring MeasuresMinutesMeasuresMinutesMeasuresMinutes ORF Individual 4ORF Individual 4ORF Individual 4 Maze Group 4Maze Group 4Maze Group 4 Adapted from B. Harn (2000)

103. Identify and Organize Resources Arrange logistics for data collection –Materials -- Who will access, prepare and organize; how will materials be organized; where will materials be stored? –Where will the assessment take place? In classroom, pod, library, cafeteria Prepare assessment “stations” (2 chairs, small desk/table). –Assign assessors to classrooms/stations. Allot time (30 minutes for 1 assessor/5 children). Specialists/volunteers -- at which school when?

104. Organize Resources Prepare assessment materials –Order assessment materials from MiBLSi Student assessment booklets - 1 per student (teacher marks errors and calculates scores) Teacher Book - 1 per assessor (student reads these passages) –Get class lists Label and collate student assessment booklets

105. Next Steps Collect Data. Enter the data into the computer. Use the data to make decisions. –Download reports. –Share data/reports with teachers. Meetings by grade Individual meetings –Make decisions about individual children. –Make system-wide decisions.

106. Team Assessment Advantages Team assessment is efficient. –5 people can assess a class in about 30 minutes. Team assessment distributes investment. Team assessment shares ownership and skills. Team assessment engages the educator in us all. Team assessment makes the results vivid and meaningful.

107. Action Planning for Reading Assessment 1.Complete the Reading Data Collection Plan and the Reading Assessment Schedule for your school. Team Time

108. TEAMWORK is not always about winning, but working together to make things happen.