How Do You Know Your Instruction is Working: An Introduction to Progress Monitoring Summer Manos FCCPS RTI Specialist 3/21/12

Our Goals for Today Our Goals for Today 1. What is Progress Monitoring? 2. Why is it important? 3. How to do I do it? 4. What resources are available to me?

What is Progress Monitoring? Frequent assessments that occur during learning- NOT after (CBM’s and CBA’s) A way to quantify a student’s rate of growth toward defined instructional goals A method that evaluates the effects of instruction/interventions on student performance and allows for changes

Who do we Progress Monitor? A closer look at a MTSS (RTI framework) Addl. Diagnostic Assessment Instruction Results Monitoring Individual Diagnostic Individual Intensive weekly All Students at a grade level Fall Winter Spring Universal Screening None Continue With Core Instruction Grades Classroom Assessments Benchmarks Group Diagnostic Small Group Differen- tiated By Skill 2 times/month Step 1 Step 2Step 3Step 4 Supplemental 1-5% 5-10% 80-90% Core Intensive

How Does Progress Monitoring Inform our Intervention? Match intervention to problem Humans tend to employ interventions with which they are comfortable instead of intervention that the student needs Intervention should be developed with the expectation that it will be altered in some way as a result of the progress monitoring data No intervention works all of the time for every student

Benefits for the Teacher Results can guide individual instruction Teachers are able to make objective decisions based on data. Results may be easily communicated to parents and staff using charts and graphs. Additional information for SBT

Curriculum-Based Measurement Specific type of Progress Monitoring A measure of student growth in basic skill areas (reading, math, and writing) Reliable and valid tests sensitive to measuring student improvement Easy and brief to administer (e.g., 1-8 minutes) Standardized administration Student data can be compared to national or local norms or criterion standards

Curriculum-Based Measurement Used for screening all students and/or monitoring individual Multiple probes of equal difficulty within each basic skill area CBMs sample basic skills across the entire grade- level curriculum Results are graphed to make instructional decisions

Why Graph? Provides a visual representation of a large amount of data A visual representation of student’s acquisition of skills and allows for easier analysis of progress Paper and pencil or electronic based graphing

Research tells us… Progress monitoring and charting are components of formative evaluation Allows you to “determine the effectiveness of an intervention during implementation so that it can be modified or changed to increase the likelihood that intended results will be achieved.” (Deno, 2002) If we use research validated reading practices, monitor student progress and make changes to instruction based on what we find, between % of children can become proficient readers. (Torgensen, 2000)

Data Based Instructional Decisions Essential for four reasons There is no guarantee that interventions will be successful, thus the intervention must be “tested” to evaluate effectiveness Increased emphasis of specific outcomes for students, data base must be generated to guide intervention decision making Pre/post testing has be shown to be unreliable (small amount of data) and provides too little data to allow for instructional decision making – progress monitoring allows for evaluation of level of performance and rate of learning Research has shown that progress monitoring is associated with improved educational outcomes

Research has shown that it works! Treatment/InterventionEffect Size Special Education Placement-.14 to.29 Modality Matched Instruction (Auditory)+.03 Modality Matched Instruction (Visual)+.04 Curriculum-Based Instruction/ Graphing and Formative Evaluation +.70 Curriculum-Based Instruction, Graphing, Formative Evaluation and Systematic use of Reinforcement +1.00

CBM Research tells us… “… that the best method of progress monitoring is Curriculum-Based Measurement.” “CBM data correlates well with high- stakes tests” McGlinchy & Hixton, (2004). Using curricular-based measurement to predict performance on state assessments in reading. School Psychology Review,33 McGlinchy & Hixton, (2004). Using curricular-based measurement to predict performance on state assessments in reading. School Psychology Review,33

Based on data Virginia (predicted) GradeFallWinterSpring R-CBM, 80% Probability of Success on State Test

Progress Monitoring Essential components that must be in place for successful progress monitoring within each tier A well-defined behavior Identification of student’s current level of performance (baseline) Instruction/Intervention Goal A measurement strategy Graph Decision-making plan

Well Defined Behavior Target behavior, observable, measurable, and specific Focus on enabling skills Skills that are prerequisite skills for more complex skills Deficiencies in enabling skills often adversely affects performance on global assessments

Skills we can measure… Reading Phonemic Awareness Alphabetic understanding Fluency Sight Words Comprehension Math Number Sense Facts Computation Applications Problem Solving

Skills Continued Written Expression Mechanics Expression Behavior Social Skills Work Completion Compliance Problem Solving Skills

CBM Procedures Scoring Reading assessments are scored number or corrects per minute Math computations are scored digits correct per minute Written expression is scored according to correct sequences

Samples of CBM Procedures Reading probes scored corrects per minute MAZE scored words correctly restored (WCR) in 3 minutes

CBM Procedures Math computations are scored by correct digits per minute (example provided by Dan Reschly)

CBM Procedures Correct Sequences for written expression Two words form a sequence, word and punctuation form a sequence. Most words and punctuation are used twice Three minutes to brainstorm, write, and edit (example provided by Tracy Hall)

How do we Progress Monitor? Step 1: Determine the area/s that needs remediation (review past data or screen) Step 2: Collect baseline data using CBM probes and set the goal line Step 3: Implement the appropriate intervention Step 4: Administer probes (4 x 4) and chart the results Step 5: Interpret results. Is the instruction working?

Goal Standard against which progress can be compared Allows for aimline to be established Possible goals Level of behavior that is expected – several ways to establish this Norms/percentile cutoffs and benchmarks Behavioral expectations Calculated growth rates – norms and benchmarks

Tier II & Tier III Progress Monitoring Charting Activity Using the data below to decide Nicole’s area of need for a reading intervention Let’s say our intervention is ‘Timed Repeated Readings’ Let’s calculate a goal for Nicole SkillNicole’s Baseline MOYBM Norms for 3 rd grade 50% EOYBM Norms for 3 rd grade 50% Oral Reading Fluency MAZE201516

Calculating Nicole’s Goal Step 1 EOYBM-MYOBM /weeks left in school year 107 – 92 ÷ 18 weeks=.83 Typical Growth.83 Average weekly improvement Step 2.83 (avg weekly improvement) x 2.0( ambitious growth rate for words per week) Most ambitious GR is 2.0 words per week.83 x 2.0 = 1.7 expected per week

Calculating Nicole’s Goal Step (targeted growth rate) x 8 (weeks of intervention) 14 total words of improvement ORF(initial oral reading fluency rate) + 14 (total words improvement) = Growth Goal 83 wpm + 14 = 97 wpm (Growth Goal)

Charting Activity Plot the baseline data for Nicole 83 WCPM Plot the goal at the end of eight weeks 97 WCPM Indicate the aimline by connecting the 2 points

Charting Activity Chart the following scores: Week 1 Thursday - 80 Week 2 Tuesday - 81 Thursday - 85 Week 3 Tuesday - 80 Make an informed decision regarding the effectiveness of the intervention “Timed Repeated Readings”

Charting Activity Decision Rules Performance trends should be analyzed periodically Trend of the data above aimline – raise the goal Trend of the data below aimline – adjust intervention If changes are made to the intervention, indicate change on the graph with a squiggle line Describe the changes on the back of the chart This allows for understanding of specific instructional adjustments that were successful/unsuccessful 4 x 4 approach

Charting Activity: Swiggle and Tweak Draw a “swiggle” to indicate a new intervention. Lets go with ‘Read Naturally’. Plot progress monitoring data for next three weeks Week 3 Thursday - 85 Week 4 Tuesday - absent Thursday - 85 Week 5 Tuesday - 90 Make an informed decision regarding the effectiveness of the intervention

Decision Rules: What is a “Good” Response to Intervention? Positive Response Gap is closing Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range Level of “risk” lowers over time Questionable Response Rate at which gap is widening slows considerably, but gap is still widening Gap stops widening but closure does not occur Poor Response Gap continues to widen with no change in rate.

Performance Time Positive Response to Intervention Expected Trajectory Observed Trajectory

Decision Rules: What is a “Questionable” Response to Intervention? Positive Response Gap is closing Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range Questionable Response Rate at which gap is widening slows considerably, but gap is still widening Gap stops widening but closure does not occur Level of “risk” remains the same over time Poor Response Gap continues to widen with no change in rate.

Performance Time Questionable Response to Intervention Expected Trajectory Observed Trajectory

Decision Rules: What is a “Poor” Response to Intervention? Positive Response Gap is closing Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range Questionable Response Rate at which gap is widening slows considerably, but gap is still widening Gap stops widening but closure does not occur Poor Response Gap continues to widen with no change in rate. Level of “risk” worsens over time

Performance Time Poor Response to Intervention Expected Trajectory Observed Trajectory

Performance Time Response to Intervention Expected Trajectory Observed Trajectory Positive Questionable Poor

Decision Rules: Linking RTI to Intervention Decisions Positive Continue intervention with current goal Continue intervention with goal increased Fade intervention to determine if student(s) have acquired functional independence.

Decision Rules: Linking RTI to Intervention Decisions Questionable Was intervention implemented as intended? If no - employ strategies to increase implementation integrity If yes -  Increase intensity of current intervention for a short period of time and assess impact. If rate improves, continue. If rate does not improve, return to problem solving.

Decision Rules: Linking RTI to Intervention Decisions Poor Was intervention implemented as intended? If no - employ strategies in increase implementation integrity If yes -  Is intervention aligned with the verified hypothesis? (Intervention Design)  Are there other hypotheses to consider? (Problem Analysis)  Was the problem identified correctly? (Problem Identification)

Decision Rules: Linking RTI to Intervention Decisions What is the number 1 reason students don’t make progress? They are not actually receiving the intervention! We need to continue to look at how we document the time and frequency of our interventions

What Resources are available for Progress Monitoring? PALS Quick Checks STAR Math STAR Reading Please consult with Reading Specialists, Math Specialist, Special Education teachers, CIRTS, RTI Specialist “The ABCs of CBM” by Hosp, Hosp & Howell