Presentation is loading. Please wait.

Presentation is loading. Please wait.

Amanda VanDerHeyden Education Research and Consulting, Inc.

Similar presentations


Presentation on theme: "Amanda VanDerHeyden Education Research and Consulting, Inc."— Presentation transcript:

1 Amanda VanDerHeyden Education Research and Consulting, Inc.
Response to Intervention: Using Data to Enhance Outcomes for all Students Amanda VanDerHeyden Education Research and Consulting, Inc.

2 16 x 3 = 48 hours

3 Data allow us to Provide faster, more effective services for ALL children Work “smarter” not harder, better utilize the talents of the school psychologist and school-based assessment and intervention teams. Make implementation SIMPLE and EASY for teachers (low cost, few errors) Prevent diagnosis

4 RTI The systematic use of assessment data to most efficiently allocate resources in order to enhance learning for all students (Burns & VanDerHeyden, 2006).

5 Essential Elements Screening
Intervention intensity matched to child need Progress monitoring Outcomes of intervention efforts linked to service allocation decisions and program evaluation

6 Why RTI in EC/EI? Adults embrace the idea that early intervention is meant to repair and prevent future learning and behavior deficits and excesses Less learning history available Technical adequacy of measures is problematic (Neiworth & Bagnato, 1992)

7 Ideas that have Failed us
Children will “outgrow” early skill deficits Waiting/delaying intervention Retention “Transition” classes

8 Ideas that have Failed us
Providing an enriched environment is sufficient to meet the needs of all “typically-developing” children

9 Measures of Cognitive Ability Not Useful at K and 1st
Children who can detect/manipulate rhymes, phonemes, or syllables learn more quickly to read irrespective of IQ, vocab, memory, and SES (Wagner et al., 1994). Measures of cognitive ability were not useful for reaching screening decisions or allocating instructional resources (Vellutino, 1996)

10 Direct Msrs of Early Literacy were Useful
Letter-sound association and use of sounds to read words– strong predictor of reading achvmt at end of first grade (Smith, Simmons, & Kame’enui, 1998). Direct msrs of early reading skills were useful (Vellutino, 1996)

11 Science of Prevention Entering phonological ability, SES, and attention/behavior– strongly predict reading growth (Torgesen et al., 1999). Torgesen et al. (2001)- 40% of children exposed to intensive intervention were exited from special education within one year of completion ( year old children identified with LD) More than half of children provided with intervention at 1st grade performed in the average range following intervention (Vellutino, 1998).

12 Early Math Findings Children are not attaining minimal standards for competency in math by end of formal schooling. Children in poverty disproportionately represented (Griffin & Case, 1997; Starkey, Klein, & Wakeley, 2004). Early intervention repairs and prevents (Fuchs, Fuchs, & Karns, 2001; Griffin & Case, 1997; Phillips, Fuchs, Fuchs, & Hamlett, 1996).

13 Consensus to These are not new ideas to EC/EI
Prevent most reading problems by reducing the # of children who enter school with poor emergent literacy skills (oral language, print knowledge, phonological processing skills)- NRP Prevent early mathematics deficits by screening, providing intervention in early numeracy Permit school success by proactive and early training in “ready to learn” behaviors These are not new ideas to EC/EI

14 Let’s abandon the ideas that have failed us
Emphasis on Maturation Let’s intervene early Emphasis on Enriched Environment Let’s ensure sufficient opportunities to gain important skills Emphasis on Teacher and Parent self-report and checklists Let’s obtain direct measures of child performance at regular intervals and in response to intervention trials

15 Identification Accuracy
CBA + RTI Criterion ITBS WJ-R STEEP Sensitivity .76 1 .58 Specificity .89 .99 .77 Positive Predictive Power .59 .67 .44 Negative Predictive Power .95 .86 Teacher Referral .46 .33 .42 .69 .94 .85 .19 .17 .45 .97 .83 VanDerHeyden, et al., 2003

16 Team Decision-Making VanDerHeyden, Witt, & Gilbertson, 2007
1Counting only those children for whom STEEP data had been completed, 9 children were recommended for evaluation by STEEP. All 9 were subsequently recommended for evaluation by the decision-making team and 8 of these children qualified for services. 2However, 17 children were not recommended for evaluation at the team decision-making meeting based on STEEP findings, but the teams decided to evaluate 10 of these children anyway. Specifically, 3 children qualified under SLD, 1 qualified under Speech and Language Impairment, and 1 qualified under Other Health Impairment. 3In , 14 children were recommended for evaluation by STEEP. Of these 14 children, 12 were evaluated and 7 qualified for services, 1 did not qualify, and 4 cases were pending at study completion. 4106 cases were not recommended for evaluation based on their having had an adequate RTI. The team decided to evaluate 14 of these children anyway. Of these 14 children evaluated, 29% of children qualified for services, 64% did not, and 1 case was pending at the completion of this study. VanDerHeyden, Witt, & Gilbertson, 2007

17 System Outcomes Referrals reduced greater than half
% who qualify improved at 4 of 5 schools SLD down from 6% of children in district in (with baseline upward trend) to 3.5% in school year Corresponding gains on high-stakes tests (VanDerHeyden & Burns, 2005) Intervention successful for about 95 to 98% of children screened VanDerHeyden, Witt, & Gilbertson, 2007

18 Cost Reduction VanDerHeyden, Witt, & Gilbertson, 2007
For schools 1 and 2 for which there were baseline and implementation data. VanDerHeyden, Witt, & Gilbertson, 2007

19 What Proportion of Ethnicity Represented Before and After Intervention in Risk Category?
Numbers are: 18/56 for minority (18 minority in bottom 16% on probes of 56 possible minority cases) and 14/154 caucasian before intervention 4/56 for minority and 9/154 for caucasian after intervention VanDerHeyden & Witt, 2005

20 Great Implementers Follow the aimline and attend to implementation integrity Understand the variables of effective instruction and engage in contextualized assessment that is technically valid for the purposes needed AND has treatment utility Minimize meeting time and avoid “the science of strange behavior…” Provide adequate resources and space for principals to be effective instructional leaders and hold them accountable for results Evaluate quality of all programs locally and make decisions about continued use based on DATA.

21 Amanda VanDerHeyden Education Research & Consulting, Inc.
RTI in Early Childhood Amanda VanDerHeyden Education Research & Consulting, Inc.

22 CBA or Mastery Model

23 CBM or General Outcome

24 CBA versus CBM Target: Drinking from cup at snack time
Hypothesis: grasping problem Intervention: alter handle to make grasping easier, provide practice using adapted cup, reinforce correct cup use Alternative hypotheses are plausible that can be tested directly Skill related-- child cannot bring cup to mouth fluently Performance related-- child is not thirsty, does not like the drink offered, finds spilled liquid aversive

25 Was intervention effective?
Adapted Cup Baseline

26 Did you solve the problem?
Baseline Adapted Cup

27 How about now? Baseline Adapted Cup

28 Baseline Adapted Cup Preferred Liquid

29 We should Measure the behaviors we are targeting and measure whether those effects contribute to improved general outcomes Assessment has too narrowly focused on individual sub-skills generally in the form of a developmental checklist

30 Amanda VanDerHeyden Education Research and Consulting, Inc.
Response to Intervention: Using Data to Enhance Outcomes for all Students Amanda VanDerHeyden Education Research and Consulting, Inc.

31 Early Screening Identifies Children At Risk of Reading Difficulty
J 2.5 5.2 5 4 Low Risk on Early Screening Reading grade level 3 2 Notes: This slide demonstrates the power of screening assessment to predict reading outcomes through the end of fourth grade. Reading outcomes were measured at the end of each year. The measure reported here assesses a combination of reading accuracy and comprehension.  The children were administered measures of phonemic awareness and letter knowledge at the beginning of first grade, and divided into two groups: At-Risk, and Low Risk. [click] The line in red shows the progress of children who began first grade performing in the bottom 15% in phonemic awareness and letter knowledge. At the end of fourth grade, these children were reading at an average level of mid second grade. In contrast, children who began first grade with higher levels of phonemic awareness and letter knowledge and roughly equivalent levels of overall ability, finished fourth grade reading at beginning fifth grade level. 1 At Risk on Early Screening Grade level corresponding to age This Slide from Reading First Experts From Reading First

32 Early Intervention Changes Reading Outcomes
J 5.2 5 4.9 Intervention With substantial instructional intervention 4 3.2 Control With research-based core but without extra instructional intervention Low Risk on Early Screening Reading grade level 3 2.5 2 Notes: Children from the bottom 15% in phonemic awareness and letter knowledge were randomly assigned to either a control group, or a group that received more intensive reading instruction in first and second grade. [click] The dotted red line shows the progress of the children who did not receive extra instructional intervention, and you can see that improved classroom instruction produced slightly better outcomes for them than in the earlier study in the same schools. [click] However, the children who were identified by the screening tests and received substantial instructional intervention did almost as well as average children by the end of fourth grade. Improved classroom instruction will help our most at-risk children learn to read better, but most will require more intensive interventions if we expect them to read at grade level by the end of fourth grade. 1 At Risk on Early Screening Grade level corresponding to age This Slide from Reading First Experts From Reading First

33 Evolution “Wait to Fail” Costly sp ed programs not improving learning
Let’s provide services early! Costly sp ed programs not improving learning Let’s shift resources to provide services in less restrictive setting! Increasing numbers of children struggling in general ed Let’s provide help in general education! Traditional measures are de-contextualized and the constructs are problematic Let’s help children who struggle academically by measuring performance in response to certain intervention strategies and then deliver what works!

34 Rationale for System Change
% increase in SLD Level and Rate of Performance Return to General Education Lack of Certified Teachers No demonstrated instructional techniques that differentially benefit SLD Drop-out Disproportionate Representation by Ethnicity Don bersoff’s comment. Reid lyon’s comment.

35 Impetus Faster, more effective services for ALL children
Work “smarter” not harder, better utilize the talents of the school psychologist and school-based assessment and intervention teams. Make implementation SIMPLE and EASY for teachers (low cost, few errors)

36 Improved Identification Accuracy for LD
Under RTI models intervention becomes a specified, operationalized variable, thus false positive identification errors should be reduced dramatically. Removing the current reliance on teacher identification and requiring direct measures of child performance in context will enhance identification accuracy.

37 More Effective Intervention
RTI is likely to facilitate less restrictive interventions and placements for children. RTI allows school psychologists to bring their expertise to bear on assessment strategies at the classroom level and assist teachers to use data formatively to enhance their instructional programming.

38 Possible Challenges of RTI

39 Decision-making Criteria
Must be operationalized and validated through research The purpose of RTI will be critical to determining how implementation should proceed

40 New Challenges for Teams
Effective intervention delivery will depend on relevant intervention variables To be effective the intervention must have been: Properly identified Implemented with integrity and with sufficient frequency, intensity, and duration

41 Screening to Enhance Educational Progress
STEEP Model Screening to Enhance Educational Progress

42 Screening tells you How is the core instruction working?
What problems might exist that could be addressed? Most bang-for-the-buck activity Next most high-yield activity is classwide intervention at Tier 2.

43 Tier 1: Screening Screening Math Screening Writing Screening
2 minutes. Scored for Digits Correct Writing Screening 3 Minutes. Scored for Words Written Correctly Reading Screening 1 Minute. Scored for Words Read Correctly

44 Tier 1: Math Screening Math Probe: Group administered.
Materials: Worksheet consisting of a series of problems sampling the target skill(s) (e.g., sums to 5, double digit multiplication with regrouping). Timing: 2 minutes Information obtained: digits correct in two minutes.

45 Math Probe Example Total Digits: 38 Errors: 5 Digits Correct: 33

46 Tier 1: Writing Screening
Writing Probe: Group administered. Materials: story starter (e.g., If I had a million dollars…) printed at the top of a blank page. Timing: 1 minute to think, 3 minutes to write. Scoring: words written or correct word sequences in three minutes.

47 Writing Example

48 Tier 1: Reading Screening
Reading Probe: Individually administered Materials: A content-controlled reading passage. Procedure: The student reads aloud as the teacher listens and records errors. Timing: 1 minute Information obtained: words read correctly in one minute.

49 CBM Reading: Sample Scoring
TRW=63 Errors=6 CRW=58

50 Screening Guidelines Efforts at Tier 1 pay off with fewer children needing individual intervention 3 times per year, single probe Use small team of trained coaches Prepare all needed materials in a packet for each teacher Score and return within 1 week on graph Use data to generate aimlines, can be used to set benchmarks

51 Class-wide Screening

52 Feedback to Teachers

53 Tier 2: Class-wide Intervention

54 No Class-wide Problem Detected

55 Tier 2: Can’t Do/Won’t Do Assessment
3-7 minutes per child “Can’t Do/Won’t Do” Individually-administered Materials Academic material that student performed poorly during class assessment. Treasure chest: plastic box filled with tangible items.

56 Can’t Do/Won’t Do Assessment

57 Decision Rule Following Can’t Do/Won’t Do Assessment

58 Tier 3: Individual Intervention

59 Response to Intervention
Before Intervention During Intervention #Correct Avg. for his Class Each Dot is one Day of Intervention Intervention Sessions Intervention in Reading

60 Response to Intervention
Before Intervention During Intervention #Correct 15 minutes to this point to set-up steep Avg. for his Class

61 Vehicle for System Change: System-wide Math Problem
Instructional range Frustrational range Each bar is a student’s performance

62 Re-screening Indicates No Systemic Problem
Fourth Grade

63 Rest of Grade at Standard
Classroom A B C D E F

64 Spring 2003– Classroom F F

65 Teacher moved to lower grade in Fall 2003

66 Class-wide Intervention
Teacher F Mult 0-12 120 100 80 Digits Correct Two Minutes 60 40 20 10/24/2003 10/31/2003 11/7/2003 11/14/2003 11/18/2003 Weeks

67 Increased Difficulty- Intervention Continues

68 Mixed Mult/Div/Fractions Probe Classroom F
Alert that I will show more later about tracking progress toward mastery of standards

69 Students Academic Systems Behavioral Systems Dave Tilly, 2005 Any
Curriculum Area Academic Systems Behavioral Systems Intensive, Individual Interventions Individual Students Assessment-based High Intensity Of longer duration Intensive, Individual Interventions Individual Students Assessment-based Intense, durable procedures 1-5% 1-5% 5-10% 5-10% Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Students Universal Interventions All students Preventive, proactive 80-90% 80-90% Universal Interventions All settings, all students Preventive, proactive Dave Tilly, 2005

70 Class-wide Intervention
Use pair-peered practice (classwide peer tutoring, PALS) Model, Guided Practice, Independent timed practice with delayed error correction, reward contingency

71 Instructional Hierarchy
Finally, problem-solving/ application practice should occur here with a mastery level skill– Core Instruction- Not Manipulated but could be Generalization But fluency building should happen here with an instructional level skill– Intervention Focus was here Fluency To gain the steepest growth, introduction of new skills should happen here– Core Instruction- Not manipulated Acquisition

72

73 Reading classwide intervention

74 Select a Few Good Interventions to Keep it Simple
Classwide Individual Math Flash card Practice Cover copy compare Cue Cards Highlighted errors Reading Listening Preview Repeated Readings Error Correction Key Words

75

76

77 Intervention Plan- 15 Min per Day
Protocol-based classwide peer tutoring, randomized integrity checks by direct observation Model, Guide Practice, Independent Timed Practice with delayed error correction Group performance contingency Teachers encouraged to Scan papers for high error rates Do 5-min re-teach for those with high-error rates Provide applied practice using mastery-level computational skill

78 Usually the higher-level reader, reads (models) first.
Rotating high –level readers helps maintain motivation

79 Intervention Plan Class Median reaches mastery range for skill, next skill is introduced Following promising results at one site in , lead to implementation district-wide grades 1-8 for all children by

80 Class-wide Math Intervention

81 With teacher support Consider time, resources, materials
Remove skill barriers with classroom training for students classroom coaching for teachers Remove implementation barriers after use new steps follow-up supportive meetings to problem solve. frequent acknowledgment of a teacher’s efforts

82 Team recommends intervention.
Unsupported means…. Team recommends intervention. All materials have to be created from scratch Teachers sent to organize class and train No one may be an expert to help the teacher No or not enough frequent objective data to collect or interpret

83 Training Package Rational Step by step protocol Model Train students
Tell Rational Step by step protocol Show Model Do Train students Implement with guided practice Implement independently with support

84 Won’t do No reinforcement for teacher behaviors
Address Common Reasons for Resistance Time consuming Lack of materials Can’t Do prevents prevents No teacher change No child change Complex not yet fluent prevents avoidance prevents Poor management prevents Too much work Won’t do No reinforcement for teacher behaviors

85 88% of interventions are not used without support
Decisions do not always correspond to data (someone must check)

86 Tier 2 Intervention Effects

87 Class 1 at Screening

88 Class 1: Following 10 Days Intervention

89 Class 1: Following 15 Days Intervention

90 Class 2 at Screening

91 Class 2: Following 5 Days Intervention

92 Class 2: Following 10 days Intervention

93 Class 3 at Screening

94 Class 3: Following 5 days Intervention

95 Following 10 Days Intervention

96 Contextually-Relevant Comparisons and Use of Trend Data

97 5th Grade Math Intervention

98 Pre-post changes to performance detected by CBM
Instructional range Frustrational range Each bar is a student’s performance

99 Fourth Grade

100 Questions Is there a classwide problem? Is there a gradewide problem?
What’s the most efficient way to deliver intervention?

101 Students Academic Systems Behavioral Systems Dave Tilly, 2005 Any
Curriculum Area Academic Systems Behavioral Systems Intensive, Individual Interventions Individual Students Assessment-based High Intensity Of longer duration Intensive, Individual Interventions Individual Students Assessment-based Intense, durable procedures 1-5% 1-5% 5-10% 5-10% Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Students Universal Interventions All students Preventive, proactive 80-90% 80-90% Universal Interventions All settings, all students Preventive, proactive Dave Tilly, 2005

102 Tier 3 Assessment Data Same implementation support as Tier 2
Instructional level performance Error analysis (high errors, low errors, pattern) Effect of incentives, practice, easier task Verify intervention effect Same implementation support as Tier 2 Instructional-level materials; Criterion-level materials

103 Tier 3 Implement for 5-15 consecutive sessions with 100% integrity
Link to referral decision Weekly graphs to teacher and weekly generalization probes outside of classroom, supply new materials Troubleshoot implementation weekly

104 Lessons Learned Most individual interventions for reading
Standard protocols with slight modifications are best Most interventions are successful (should be successful) Generalization must be attended to Team will not follow data without support and training to do so Coordinate intervention start times with principal and stagger start dates (10-15 at a time plus Tier 2’s). Organize master schedule for data collection and Tier times

105 Tier 3 Intervention >5% of children screened (total population) IF solid Tier 1 Possibly as low as 2% IF solid Tier 1 and Tier 2 About 1-2% failed RTI; 10% of most at-risk

106 Successful RTI

107 Successful RTI

108 Successful Writing

109 Successful RTI

110 Successful Math

111 Unsuccessful Math

112 Digits Correct Two Minutes
Pass the AIMS Digits Correct Two Minutes You can do this with CBM. This cannot be accomplished with periodic benchmark assessments with set numbers of items and ceilings and scoring format of multiple choice 1 12 Weeks

113 Here is 3rd grade at Cottonwood
Here is 3rd grade at Cottonwood. Each circle corresponds to a students score on a reading CBM probe in March and the AIMS reading score the same month. So the circle near the blue lines is a child who read 158 wc/min and scored 486 on the AIMS Reading. The diagonal line represents the “best fit” line or the line closest to all the circles. This shows there is a strong positive correlation between CBM and AIMS reading scores with this group.

114 How to Set a Benchmark This is words read correctly per minute. You move this line up and down to “catch” as many of those who will not pass as possible. 431 = “pass” for AIMS. This line does not move

115 Setting 95 wc/min as the “pass” standard for CBM
These are children who were predicted to pass AIMS based on CBM and did pass. “Hits” These are the children predicted to pass AIMS who actually failed. “False Negative Errors.” The worst kind of error. These are the children predicted to fail AIMS who actually passed. “False Positive Errors.” These are the children who were predicted to fail AIMS based on CBM who did fail. “Hits”

116 Moving the horizontal line up will “catch” two more cases who failed the AIMS, but will result in many more “false positive errors.” Thus, 95 wc/min at 3rd grade at Cottonwood is a good standard that will tell you which children are likely to fail the AIMS reading section.

117 ROC printout

118 Mastery Model Measurement (CBA)
Letter naming fluency Isolated sound fluency Beginning sound fluency Ending sound fluency

119 General Outcome Measurement (CBM)
Beginning sound fluency Letter naming fluency Isolated sound fluency Ending sound fluency Words read correctly per minute

120 Tracking Year-Long Growth
mastery aimline instructional

121 Tips for Effective Implementation

122 Our Recipe for Intervention Success
PREPARE Identify and Use standard protocols for intervention Develop all needed materials Develop packets or put on a central web site Determine graphing program 122

123 Our Recipe for Intervention Success
TRAIN Explain Watch the teacher do it with the actual child before you leave Call or meet teacher after first day to problem solve 123

124 Our Recipe for Intervention Success
COLLECT DATA AND SUPPORT Each week, graph intervention performance and do a generalization check with the child. Graphed feedback to teachers with generalization checks for individual intervention once per week Response-dependent performance feedback to sustain implementation accuracy Monthly CBM to track growth and enhance existing Tier 1 Programs or advise new Tier 1 Data to principal weekly. Summarize effects and integrity of procedures. 124

125 Tracking Title Progress

126 Our Recipe for Intervention Success
USE DATA TO MAKE DECISIONS RTI successful if child performs criterion-level probe (from screening) in the instructional range. RTI unsuccessful if 15 consecutive intervention sessions and criterion probe is not in the instructional range. Increase task difficulty for intervention if child scores at mastery on task during intervention sessions 126

127 Infrastructure for Implementation
Grade-level planning periods can be utilized Special education “team” at school can be utilized School Psych must be on-site 1 day/week Developing master schedule for Tier 1, 2, and 3 intervention times is useful Integrate efforts with evaluation referral team efforts (consider major reduction in meeting time and shift to intervention efforts!)

128 Materials needed Computer and software to organize data
Student data imported. Clerical person to enter data on-site for tier 1 screen only. Color printer to print graphs + extra color cartridges Probe materials, digital count-down timers Intervention protocols, intervention materials (e.g., flashcard sets, reading materials) Access to copier and some assistance with copying Reinforcers for treasure chest (no more than $500 per school)

129 Guidelines for Implementers
Use single trial scores for screening Following screening, grade-wide graphs to principal Return data to teachers within 48 hours with personal interpretation at grade-level team meeting Include principal in critical meetings Involve teachers at all stages 129

130 Guidelines for Implementers
Learn about curriculum and instruction. Integrate RTI with ongoing school and system reform efforts Thoughtfully merge to subtract duplicate activities and to enhance more comprehensive supplemental and core instructional support activities that may be in place Use RTI data to evaluate the value of ALL instructional programs or resource allocation decisions. Quantify bang for the buck using student performance data.

131 Lessons Learned from Vail USD
Infrastructure for education being a results-based enterprise Accountability Principal is the Instructional Leader of a school Principal as change agent School psychologist as change agent Replace resources/substitute don’t add Minimize meetings Track outcomes that matter

132 Great Instructional Leaders
Have a filter Allocate time and resources according to their filter Use an AIMLINE Have a framework for making data-driven decisions (know how to access the data they need to reach timely decisions) Hold teachers, staff, students accountable Research findings on effective schooling

133 Great School Psychologists
Hand the principal the data the principal did not know to ask for but can’t live without Follow the aimline and attend to implementation integrity Understand the variables of effective instruction and engage in contextualized assessment that is technically valid for the purposes needed AND has treatment utility Minimizes meeting time and “avoids the science of strange behavior…”

134 Great Districts Minimize time away from school, but use time together to review school improvement implementation efforts and ongoing results Have the will to proactively chart the course of a district Provide adequate resources and space for principals to be effective instructional leaders and hold them accountable for results Respect the role of parents and actively engage them Have a framework for evaluating results (know how to access data for decision making) Evaluate quality of all programs locally and make decisions about continued use based on DATA.

135 Great Teachers Use data to identify where more/different/less instruction is needed Have as a goal to accelerate all learning of all children Proactively address barriers to learning Take responsibility for learning that occurs in the classroom Are confident and ready to collaborate in the classroom Appreciate childhood and children (a little humor, lots of patience, enthusiasm)

136 How To Do Classwide Intervention within RTI
Amanda VanDerHeyden Education Research and Consulting, Inc.

137

138

139 In Low-Achieving Classrooms (more than 50% of class scored in the frustrational range on probes)
STEEP Teacher Referral Sensitivity .75 .55 Specificity .88 .68 Positive Predictive Power .69 .35 Negative Predictive Power .91 .82 VanDerHeyden & Witt, 2005

140 In Average to High-Achieving Classrooms (less than 20% of class scored in frustrational range on probes) STEEP Teacher Referral Sensitivity .67 Specificity 1.0 Positive Predictive Power Negative Predictive Power .97 .95 VanDerHeyden & Witt, 2005

141 Identification Accuracy
High-achieving classrooms (<20%) Procedures paired with RTI criterion were more accurate than other commonly used screening devices Low-achieving classrooms (>50%) VanDerHeyden & Witt, 2005

142 Research Finding VanDerHeyden, Broussard, et al. (2004).
Prior to single instructional session, children receiving special ed services were significantly lower performing on math probes. Following single instructional session, no significant difference was observed.

143 Mixed Mult/Div/Fractions Probe Classroom F
Alert that I will show more later about tracking progress toward mastery of standards

144 Sore Thumb Test

145 Response to Classwide Intervention
Student A

146 Individual Math Intervention Can’t Do Problem

147 Successful Math Intervention
Baseline Intervention Novel, grade-level probe

148 Why do Classwide Intervention
Efficiency Accuracy Efficacy

149 Rest of Grade at Standard
Classroom A B C D E F

150 Using Screening Data to Identify Class-wide and System-wide Instructional Problems

151 Consider The Task Integrity of Administration Reliability of Scoring
Use software to organize the data

152 Mult 0-9 4th Grade Fall Screening

153 Mult/Div/Fractions 4th Grade Winter

154 Grade-wide Data

155

156

157 3rd Grade Mult 0-9 Spring

158 Guided Practice

159 Fourth Grade Reading Level: Math Skill 1: Math Skill 2:
Guide audience to pick the skill

160 Questions Is there a classwide problem? Is there a gradewide problem?
What’s the most efficient way to deliver intervention?

161

162

163

164

165 What Data do you Want for Principal?

166

167

168

169 Questions Is there a classwide problem? Is there a gradewide problem?
What’s the most efficient way to deliver intervention?

170 Independent Practice

171 First Grade Reading What do you want to know?
Is there a class-wide problem? Is there a grade-wide or systemic problem? What’s the most efficient way to deliver intervention? (whole class, small group, individual) What is the next step for Class 1, 2, 3, 4?

172 Class 1

173 Class 2

174 Class 3

175 Class 4 Growth trajectory basis for 1st grade. Have to select a “criterion” to use at your site. The closer the median score is to the instructional minimum, the more work for the school psychologist!

176 Grade-wide Data

177

178 Class 4

179 Prepare for Training Day
Locate probes (e.g., worksheet factory, intervention central, basic skill builders) Identify Graphing Program (excel) Locate or develop scripts (gosbr.net; interventioncentral.org) Develop quick access to materials for teachers Determine integrity monitoring plan Identify common time for intervention Set start date

180 Set a daily routine. Time, location of materials, process for weekly assessment. Set a date and time for 30-min training Set a date for a later 15-minute first practice time with teacher

181 Materials needed Computer and software to organize data
Student data imported. Clerical person to enter data on-site for tier 1 screen only. Color printer to print graphs + extra color cartridges Probe materials, digital count-down timers Intervention protocols, intervention materials (e.g., flashcard sets, reading materials) Access to copier and some assistance with copying Reinforcers for treasure chest (no more than $500 per school)

182 Measurement Plan Weekly probe of Intervention skill
Weekly probe of Retention of previously mastered computational skills Monthly probe using GOM approach to monitor progress toward year-end computational goals To this you might add an application measure

183 Intervention Plan Class Median reaches mastery range for skill, next skill is introduced Following promising results at one site in , lead to implementation district-wide grades 1-8 for all children by

184 Rationale Provided teacher/student a script that tells….
what the student has to do and when what the teacher should do to support student how the student will know how he/she is doing Treatment considerations for integrity issues All steps are clearly needed Includes lots of student response opportunities Disrupts class as little as possible Requires little teacher time ( < 15 min/day) Considers resources to decrease teacher effort Used simple language All the materials are available

185 Trainer Observe the teacher using the steps on the intervention script 2. Check off steps used. 3. Prompt the teacher to do any missed step . 4. Problem Solve any noted “blockers” 5. Continue until accurately implemented without prompts

186 Why verbal and modeling training alone do not work:
No instructions when problems arise In adequate classroom management prevents interferes Low implementation Not enough child assistance for bx change Non-specific steps prevents prevents Inadequate materials interferes Low frequencies Lack of practice with feedback Lack of reinforcement For teacher behaviors

187 Math Partners Progress Chart
Count every digit that is not circled. This is your score! Write your score on your math sheet. Find today’s date on this page and write your score on the line. Put a Star on the graph to mark today’s score.

188 Weekly Progress Monitoring
Administer classwide math worksheet Target skill once per week Criterion skills periodically to monitor growth Use incentives to maximize performance Apply decision rules

189 Progress Review Review folders to ensure that intervention was used correctly for at least 4 days that week If this is not the case, conduct another in-class training day. Graph weekly progress monitoring assessment data

190 Decision making Review data to make decisions:
DATA OUTCOME 1: Class median is below mastery range and most students gaining digits correct per week. ACTION: Consider implementing intervention for an additional week and then review progress again.

191 Decision making DATA OUTCOME 2: Class median is below mastery range and most students are not gaining digits correct per week: ACTION: Check Integrity first and address with training if needed. Consider implementing intervention for an additional week with incentives or easier task and then review progress again.

192 Decision making DATA OUTCOME 3: If the class median is above mastery range then consider: ACTION: Increasing task difficulty and continuing classwide intervention.   ACTION: For students performing in the frustration range, consider Tier 3 assessment and intervention.

193 Troubleshoot Intervention Support Yes No
Was the intervention developed to ensure that it required minimal classroom time and resources and fit within daily classroom routines? Are materials readily available to the teacher? Was a step-by-step “coach card” provided? Was the teacher shown how to implement the intervention by a “coach?” Did the coach observe implementation of the intervention to ensure that the teacher could use the intervention correctly and had all needed materials? Was weekly follow-up support provided to the teacher after initial training? Are integrity data graphed to show used correctly? Is an administrator involved?

194 Results

195 Tier 1 Screening Indicates Class-wide Problem

196 Tier 2: Class-wide Intervention
Teacher F Mult 0-12 120 100 80 Digits Correct Two Minutes 60 40 20 10/24/2003 10/31/2003 11/7/2003 11/14/2003 11/18/2003 Weeks

197 Contextually-Relevant Comparisons and Use of Trend Data

198

199 Pre-post changes to performance detected by CBM
Instructional range Frustrational range Each bar is a student’s performance

200 Fourth Grade

201 Additional Research Questions
What level of performance predicted strongest subsequent growth given intervention? What level of performance predicted skill would be retained about 3 months after it was taught? Did mastery of foundation skills shorten the number of trials required to master more complex related skills?

202 What level of performance predicted strongest subsequent growth given intervention?
Across 4 weeks of intervention (4 datapoints) OLS used to estimate slope Children achieving slopes equal to or greater than the 66th percentile were identified as strong responders Starting fluency (prior to intervention) was identified for the group of strong responders and range was estimated as average starting fluency +/- 1 standard deviation Tested new criterion on second set of scores Burns & VanDerHeyden, 2006

203 General Findings Growth rates and trials to criterion varied dramatically across skills Retention probe was strongest predictor of year-end SAT-9 performance Mastery level performance on early skills predicted fewer trials to criterion on future related complex skills Fluency scores higher than “mastery” predicted retention of skill over time (about +20 dc/min) VanDerHeyden & Burns, 2008; VanDerHeyden & Burns, in submission

204 Identification Accuracy
CBA + RTI Criterion ITBS WJ-R STEEP Sensitivity .76 1 .58 Specificity .89 .99 .77 Positive Predictive Power .59 .67 .44 Negative Predictive Power .95 .86 Teacher Referral .46 .33 .42 .69 .94 .85 .19 .17 .45 .97 .83 VanDerHeyden, et al., 2003

205 Percent Identified at each Tier
CBM (Classwide Assessment) 55 (15%) CBM + Reward (Performance/skill Deficit Assessment) 40 (11%) CBM + Reward + Instruction (STEEP +) 22 (6%) Teacher Referral 32 (19%) CIBS-R 64 (18%) DRA 17 (9%) RTI Criterion Assessment 17 (5%) WJ-R 12 ITBS deficit 3 (4%) VanDerHeyden, et al., 2003

206 Students Academic Systems Behavioral Systems Dave Tilly, 2005 Any
Curriculum Area Academic Systems Behavioral Systems Intensive, Individual Interventions Individual Students Assessment-based High Intensity Of longer duration Intensive, Individual Interventions Individual Students Assessment-based Intense, durable procedures 1-5% 1-5% 5-10% 5-10% Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Targeted Group Interventions Some students (at-risk) High efficiency Rapid response Students Universal Interventions All students Preventive, proactive 80-90% 80-90% Universal Interventions All settings, all students Preventive, proactive Dave Tilly, 2005

207 To work smart, we must ask
What is the purpose of our assessment? How do we know it serves our purpose? Is this the cheapest way to do it?

208 Our Goal Collect the best information in the shortest possible period of time

209 Strategy in a Nutshell Identify the goal (DV’s)
Behavior to increase (fluency, comprehension) Behavior to decrease (errors) Match the strategy to the goal (Daly et al., 1996) Monitor the DV’s and the IV’s (intervention variables)

210 Find instructional level (sampling back)
Identify the “root” of the problem (e.g., division is difficult because subtraction is not fluent or multiplication is not fluent; poor decoding skills v. dolce words; production v. accuracy in writing) This is the most important part of the process

211 Define the Behaviors/skills
Pronounce beginning word sounds Define the Behaviors/skills Fluent Letter Sound Production Accurate Letter Sound Production Association of Letters with phonemes Fluent Letter Naming Accurate Letter Naming

212 Identify Reinforcers and Logical Consequences
Use a treasure chest Use an activity survey or reinforcer checklist Use incidental teaching strategy Use logical or natural consequences

213 Measure Baseline Performance and Set Goals

214 Functional Assessment
What is an effective intervention?

215 Functional Assessment

216 Functional Assessment
BL Intervention Performance Feedback

217 Troubleshooting Intervention Effects at Tier 3

218 Teachers must weigh the following
What outcomes does completing work produce? Positive feedback from the teacher Positive attention from peers, status Access to fun activities or reinforcement Avoidance of punitive consequences What outcomes does not completing work produce? Escape from assignment, from classroom setting Peer attention Adult attention (even if it is negative). Some students are so motivated to obtain adult attention that it does not matter if the attention is negative or positive.

219 Antecedent Variables Task Difficulty, Sequencing of Skills
Time actively engaged in learning (AET) Opportunities to respond Other lesson variables (pacing, exemplars) Behaviors interfering with instruction (teacher and child)

220 Task Difficulty, Sequencing
Sample back measuring fluency of performance on basic skills The idea is to identify the weak point in the chain Define the target skill for intervention and the criterion skill (goal)

221 Academic Engaged Time Impacts opportunities to respond
Robust predictor of achievement Average 2nd grader (Rosenshine) spent less than 1 hour AET per day. Check transitions, classroom management, time allocated for independent practice, active monitoring/scanning

222 Other Lesson Variables
Presentation of materials and Sequencing of Lesson Organized Clear, redundant examples Exemplars sufficient S+ and S- Checking for student understanding Pacing of lesson

223 Behaviors Interfering with Instruction/Intervention
Teacher behaviors Implementation accuracy and consistency Fuchs & Fuchs, 1987; Gresham, 1991; Happe, 1982; Wickstrom, Jones, LaFleur, & Witt, 1998 Teacher understanding/adequately trained Train to fluency criterion (Chandler, Lubeck, & Fowler, 1992) Teacher acceptability of intervention (prospective, ongoing, link to changes) Adequate resources to conduct intervention

224 Behaviors Interfering with Instruction
Child behaviors Disruptive or inattentive behaviors Concurrent options available (access to reinforcing outcomes by not completing intervention) Consider can’t do/won’t do (although programming for motivation is important anyway)

225 Consequences Reinforcing consequences (for correct and incorrect performance) Escaping task Extra attention (staying in at recess may be reinforcing) Feedback Frequency Immediacy Accuracy Correct error immediately, have student repeat response correctly, match response to instructional situation & learner (Heubusch & Lloyd, 1998)

226 Tier 3 Intervention >5% of children screened (total population) IF solid Tier 1 Possibly as low as 2% IF solid Tier 1 and Tier 2 About 1-2% failed RTI; 10% of most at-risk VanDerHeyden et al., 2007

227 Tier 3 Findings Most interventions for reading Math is next
Math is at least two-dimensional--- computational and operational fluency plus application or conceptual understanding Most interventions are not implemented well and that’s why they fail Tier 3 interventions are likely to occur on below grade level tasks AND require acquisition-type instruction (discrimination training to establish accurate responding)

228 Successful Math Intervention

229 Unsuccessful Math Intervention

230 Integrity Matters 59% Integ 96% Integrity

231 Integrity Matters

232 Integrity Matters

233 Integrity Untreated integrity problems become student learning deficits, schoolwide learning problems, and false positive decision errors Integ problems affect dose and quality of the treatment (an intervention implemented with fidelity is a functionally different intervention than one implemented inconsistently Integ positively correlated with student learning gains, amount of intervention covered Even veteran sites require monitoring and follow-up

234 Tips for Effective Implementation

235 Our Recipe for Intervention Success
PREPARATION Identify and Use standard protocols for intervention Develop all needed materials Develop packets or put on a central web site Determine graphing program 235

236 Our Recipe for Intervention Success
TRAIN Explain Watch the teacher do it with the actual child before you leave Call or meet teacher after first day to problem solve 236

237 Our Recipe for Intervention Success
DATA COLLECTION AND SUPPORT Each week, graph intervention performance and do a generalization check with the child. Graphed feedback to teachers with generalization checks for individual intervention once per week Response-dependent performance feedback to sustain implementation accuracy Monthly CBM to track growth and enhance existing Tier 1 Programs or advise new Tier 1 Data to principal weekly. Summarize effects and integrity of procedures. 237

238 Our Recipe for Intervention Success
DATA DECISION -MAKING RTI successful if child performs criterion-level probe (from screening) in the instructional range. RTI unsuccessful if 15 consecutive intervention sessions and criterion probe is not in the instructional range. Increase task difficulty for intervention if child scores at mastery on task during intervention sessions 238

239 Guidelines for Implementers
Use single trial scores for screening Following screening, grade-wide graphs to principal Return data to teachers within 48 hours with personal interpretation at grade-level team meeting Include principal in critical meetings Involve teachers at all stages 239

240 Guidelines for Implementers
Learn about curriculum and instruction. Integrate RTI with ongoing school and system reform efforts Thoughtfully merge to subtract duplicate activities and to enhance more comprehensive supplemental and core instructional support activities that may be in place Use RTI data to evaluate the value of ALL instructional programs or resource allocation decisions. Quantify bang for the buck using student performance data.

241 Infrastructure for Implementation
Grade-level planning periods can be utilized Special education “team” at school can be utilized School Psych must be on-site 1 day/week Developing master schedule for Tier 1, 2, and 3 intervention times is useful Integrate efforts with evaluation referral team efforts (consider major reduction in meeting time and shift to intervention efforts!) Use existing instructional periods to target student needs more effectively See NASDSE blueprint for implementation Brown-Chidsey book coming from Guilford

242 Resources www.naspweb.org- early childhood position statements
Best Practices Big Math for Little Kids Number Worlds Headsprout Kansas Center

243 For More Information amandavande@gmail.com www.isteep.com
Thank you to the US Dept of Education for providing all film clips shown in this presentation 243


Download ppt "Amanda VanDerHeyden Education Research and Consulting, Inc."

Similar presentations


Ads by Google