Presentation on theme: "Rate of Improvement Calculation and Decision Making"— Presentation transcript:
1Rate of Improvement Calculation and Decision Making Caitlin S. Flinn, EdS, NCSPAndrew E. McCrea, MS, NCSP
2Why we’re here…While there exists a wealth of convincing research supporting the implementation of a response-to-intervention (RtI) framework, there are many questions yet to be empirically answered.Within multi-tiered model of assessment and instruction/intervention, how do we know whether a student is learning?
3Measuring Learning Class tests Quizzes Assignment/homework completion and accuracyAsk students questions in classGrades/report cardsState/local assessmentsUniversal screening, benchmark assessmentsProgress monitoring
4With Progress Monitoring Data… How do we know if a student is learning?Look at the data pointsWhere are they on the graph?Are the data points getting closer to the goal or benchmark?Is there a way to measure growth?Make an aimline toward goalLook to see where data points are compared to aimlineCalculate Rate of Improvement (RoI)
5Today’s ObjectivesExplain what RoI is, why it is important, and how to compute it.Establish that Simple Linear Regression should be the standardized procedure for calculating RoI.Discuss how to use RoI within a problem solving/school improvement model.
6RoI DefinitionRate of Improvement can be described algebraically as the slope of a lineSlope is defined as: the vertical change over the horizontal change on a Cartesian plane. (x-axis and y-axis graph)Also called: Rise over runFormula: m = (y2 - y1) / (x2 - x1)Describes the steepness of a line (Gall & Gall, 2007)
7RoI DefinitionFinding a student’s RoI is determining the student’s learningCreating a line that fits the data points, a trendlineTo find that line, we use:Linear regressionOrdinary Least Squares
8How does Rate of Improvement Fit into the Larger Context?
9School Improvement/Comprehensive School Reform Response to InterventionDual Discrepancy: Level & GrowthRate of Improvement
10School Improvement/ Comprehensive School Reform Grade level content expectations (ELA, math, science, social studies, etc.).Work toward these expectations through classroom instruction.Understand impact of instruction through assessment.
11Assessment Formative Assessments/High Stakes Tests Does student have command of content expectation (standard)?Universal Screening using CBMDoes student have basic skills appropriate for age/grade?
12AssessmentQ: For students who are not proficient on grade level content standards, do they have the basic reading/writing/math skills necessary?A: Look at Universal Screening; if above criteria, intervention geared toward content standard, if below criteria, intervention geared toward basic skill.
13Progress MonitoringFrequent measurement of knowledge to inform our understanding of the impact of instruction/intervention.Measures of basic skills (CBM) have demonstrated reliability & validity (see table at
15So…Rate of Improvement (RoI) is how we understand student growth (learning).RoI is reliable and valid (psychometrically speaking) for use with CBM data.RoI is best used when we have CBM data, most often when dealing with basic skills in reading/writing/math.RoI can be applied to other data (like behavior) with confidence too!RoI is not yet tested on typical Tier I formative classroom data.
16RoI is usually applied to… Tier One students in the early grades at risk for academic failure (low green kids).Tier Two & Three Intervention Groups.Special Education Students (and IEP goals)Students with Behavior PlansLove the tractor
17RoI Foundations Deno, 1985 Curriculum-based measurement General outcome measuresTechnically adequateShortStandardizedRepeatableSensitive to change
18RoI Foundations Fuchs & Fuchs, 1998 Hallmark components of Response to InterventionOngoing formative assessmentIdentifying non-responding studentsTreatment fidelity of instructionDual discrepancy modelOne standard deviation from typically performing peers in level and rate
19RoI Foundations Ardoin & Christ, 2008 Slope for benchmarks (3x per year)More growth from fall to winter than winter to springMight be helpful to use RoI for fall to winterAnd a separate RoI for winter to spring
20RoI Foundations Fuchs, Fuchs, Walz, & Germann, 1993 Typical weekly growth rates in oral reading fluency and digits correctNeeded growth to remediate skillsStudents who had 1.5 to 2.0 times the slope of typically performing peers were able to close the achievement gap in a reasonable amount of time
21RoI Foundations Deno, Fuchs, Marston, & Shin, 2001 Slope of frequently non-responsive children approximated slope of children already identified as having a specific learning disability
22How many data points?10 data points are a minimum requirement for a reliable trendline (Gall & Gall, 2007)Is that reasonable and realistic?How does that affect the frequency of administering progress monitoring probes?How does that affect our ability to make instructional decisions for students?
23How can we show RoI?Speeches that included visuals, especially in color, improved recall of information (Vogel, Dickson, & Lehman, 1990)“Seeing is believing.”Useful for communicating large amounts of information quickly“A picture is worth a thousand words.”Transcends language barriers (Karwowski, 2006)Responsibility for accurate graphical representations of data
24Skills for Which We Compute RoI ReadingOral Reading FluencyWord Use FluencyReading ComprehensionMAZERetellEarly Literacy SkillsInitial SoundLetter NamingLetter SoundPhoneme SegmentationNonsense WordSpellingWritten ExpressionBehaviorMathMath ComputationMath FactsEarly NumeracyOral CountingMissing NumberNumber IdentificationQuantity Discrimination
25Guidelines? Visual inspection of slope Multiple interpretations Instructional servicesNeed for explicit guidelines
26Ongoing ResearchRoI for instructional decisions is not a perfect processResearch is currently addressing sources of error:Christ, 2006: standard error of measurement for slopeArdoin & Christ, 2009: passage difficulty and variabilityJenkin, Graff, & Miglioretti, 2009: frequency of progress monitoring
27Future Considerations Questions yet to be empirically answeredWhat parameters of RoI indicate a lack of RtI?How does standard error of measurement play into using RoI for instructional decision making?How does RoI vary between standard protocol interventions?How does this apply to non-English speaking populations?
29Multiple Methods for Calculating Growth Visual Inspection Approaches“Eye Ball” ApproachSplit Middle ApproachTukey MethodQuantitative ApproachesLast point minus First point ApproachSplit Middle & Tukey “plus”Linear Regression Approach
34Tukey Method Divide scores into 3 equal groups Divide groups with vertical linesIn 1st and 3rd groups, find median data point and median week and mark with an “X”Draw line between two “Xs”(Fuchs, et. al., Summer Institute Student progress monitoring for math.
42Any Method of Visual Inspection RoI Consistency?Any Method of Visual Inspection???Last minus First0.75Split Middle “Plus”0.63Tukey “Plus”Linear Regression1.10
43RoI Consistency?If we are not all using the same model to compute RoI, we continue to have the same problems as past models, where under one approach a student meets SLD criteria, but under a different approach, the student does not.Hypothetically, if the RoI cut-off was 0.65 or 0.95, different approaches would come to different conclusions on the same student.
44Difference in RoI b/w LmF & LR Methods RoI Consistency?Last minus First (Iris Center) and Linear Regression (Shinn, etc.) only quantitative methods discussed in CBM literature.Study of 37 at risk 2nd graders:Difference in RoI b/w LmF & LR MethodsWhole Year0.26 WCPMFall0.31 WCPMSpring0.24 WCPMMcCrea (2010) Unpublished data
45Technical AdequacyWithout a consensus on how to compute RoI, we risk falling short of having technical adequacy within our model.
47Literature shows that Linear Regression is Best Practice Student’s daily test scores…were entered into a computer program…The data analysis program generated slopes of improvement for each level using an Ordinary-Least Squares procedure (Hayes, 1973) and the line of best fit.This procedure has been demonstrated to represent CBM achievement data validly within individual treatment phases (Marston, 1988; Shinn, Good, & Stein, in press; Stein, 1987).Shinn, Gleason, & Tindal, 1989
48Growth (RoI) Research using Linear Regression Christ, T. J. (2006). Short-term estimates of growth using curriculum based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence intervals. School Psychology Review, 35,Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using curriculum based measurement to establish growth standards for students with learning disabilities. School Psychology Review, 30,Good, R. H. (1990). Forecasting accuracy of slope estimates for reading curriculum based measurement: Empirical evidence. Behavioral Assessment, 12,Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22,
49Growth (RoI) Research using Linear Regression Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM progress monitoring. Exceptional Children, 75,Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for curriculum-based measurement. The Journal of Special Education, 23,Shinn, M. R., Good, R. H., & Stein, S. (1989). Summarizing trend in student achievement: A comparison of methods. School Psychology Review, 18,
50So, Why Are There So Many Other RoI Models? Ease of applicationFocus on Yes/No to goal acquisition, not degree of growthHow many of us want to calculate OLS Linear Regression formulas (or even remember how)?
51Pros and Cons of Each Approach Eye BallEasyUnderstandableSubjectiveSplit Middle & TukeyNo software neededCompare to Aim/Goal lineYes/No to goal acquisitionNo statistic provided, no idea of the degree of growth
52Pros and Cons of Each Approach Last minus FirstProvides a growth statisticEasy to computeDoes not consider all data points, only twoSplit Middle & Tukey “Plus”Considers all data points.No support for “plus” part of methodologyLinear RegressionAll data pointsBest PracticeCalculating the statistic
54Get Out Your Laptops!Open Microsoft ExcelI loveROI
55Graphing RoI For Individual Students Programming Microsoft Excel to Graph Rate of Improvement:Fall to Winter
56Setting Up Your Spreadsheet In cell A1, type 3rd Grade ORFIn cell A2, type First SemesterIn cell A3, type School WeekIn cell A4, type BenchmarkIn cell A5, type the Student’s Name (Swiper Example)
57Labeling School WeeksStarting with cell B3, type numbers 1 through 18 going across row 3 (horizontal).Numbers 1 through 18 represent the number of the school week.You will end with week 18 in cell S3.Why 18?That’s half of a school year.
58Labeling DatesNote: You may choose to enter the date of that school week across row 2 to easily identify the school week.
59Entering Benchmarks (3rd Grade ORF) In cell B4, type 77. This is your fall benchmark.In cell S4, type 92. This is your winter benchmark.
60Entering Student Data (Sample) Enter the following numbers, going across row 5, under corresponding week numbers.Week 1 – 41Week 8 – 62Week 9 – 63Week 10 – 75Week 11 – 64Week 12 – 80Week 13 – 83Week 14 – 83Week 15 – 56Week 17 – 104Week 18 – 74
61*CAUTION*If a student was not assessed during a certain week, leave that cell blankDo not enter a score of Zero (0) it will be calculated into the trendline and interpreted as the student having read zero words correct per minute during that week.
62Graphing the Data Highlight cells A4 and A5 through S4 and S5 Follow Excel 2003 or Excel 2007 directions from here
63Graphing the Data Excel 2003 Excel 2007 Across the top of your worksheet, click on “Insert”In that drop-down menu, click on “Chart”Excel 2007Click InsertFind the icon for LineClick the arrow below Line
64Graphing the Data Excel 2003 Excel 2007 A Chart Wizard window will appearExcel 20076 graphics appear
65Graphing the Data Excel 2003 Excel 2007 Choose “Line” Choose “Line with markers…”Excel 2007Choose “Line with markers”
66Graphing the Data Excel 2003 Excel 2007 “Data Range” tab “Columns” Your graph appears
67Graphing the Data Excel 2003 Excel 2007 “Chart Title” “School Week” X Axis“WPM’ Y AxisExcel 2007Change your labels by right clicking on the graph
68Graphing the Data Excel 2003 Excel 2007 Choose where you want your graphExcel 2007Your graph was automatically put into your data spreadsheet
69Graphing the Trendline Excel 2003Right click on any of the student data pointsExcel 2007
70Graphing the Trendline Excel 2003Choose “Linear”Excel 2007
71Graphing the Trendline Excel 2003Choose “Custom” and check box next to “Display equation on chart”Excel 2007
72Graphing the Trendline Clicking on the equation highlights a box around itClicking on the box allows you to move it to a place where you can see it better
73Graphing the Trendline You can repeat the same procedure to have a trendline for the benchmark data pointsSuggestion: label the trendline Expected ROIMove this equation under the first
75Individual Student Graph The equation indicates the slope, or rate of improvement.The number, or coefficient, before "x" is the average improvement, which in this case is the average number of words per minute per week gained by the student.
76Individual Student Graph The rate of improvement, or trendline, is calculated using a linear regression, a simple equation of least squares.To add additional progress monitoring/benchmark scores once you’ve already created a graph, enter additional scores in Row 5 in the corresponding school week.
77Individual Student Graph The slope can change depending on which week (where) you put the benchmark scores on your chart.Enter benchmark scores based on when your school administers their benchmark assessments for the most accurate depiction of expected student progress.
78Programming Excel First Semester Calculating Needed RoICalculating Benchmark RoICalculating Student’s Actual RoI
79Quick Definitions Needed RoI Benchmark RoI Student’s Actual RoI The rate of improvement needed to “catch” up to the next benchmark.Benchmark RoIThe rate of improvement of typically performing peers according to the normsStudent’s Actual RoIBased on the available data points, this is the student’s actual rate of improvement per week
80Calculating Needed RoI In cell T3, type Needed RoIClick on cell T5In the fx line (at top of sheet) type this formula =((S4-B5)/18)Then hit enterYour result should read:This formula simply subtracts the student’s actual beginning of year (BOY) benchmark from the expected middle of year (MOY) benchmark, then dividing by 18 for the first 18 weeks (1st semester).
81Calculating Benchmark RoI In cell U3, type Benchmark RoIClick on cell U4In the fx line (at top of sheet) type this formula =SLOPE(B4:S4,B3:S3)Then hit enterYour result should read:This formula considers 18 weeks of benchmark data and provides an average growth or change per week.
82Calculating Student Actual RoI Click on cell U5In the fx line (at top of sheet) type this formula =SLOPE(B5:S5,B3:S3)Then hit enterYour result should read:This formula considers 18 weeks of student data and provides an average growth or change per week.
83Graphing RoI For Individual Students Programming Microsoft Excel to Graph Rate of Improvement:Winter to Spring
84Setting Up Your Spreadsheet In cell A1, type 3rd Grade ORFIn cell A2, type Second SemesterIn cell A3, type School WeekIn cell A4, type BenchmarkIn cell A5, type the Student’s Name (Swiper Example)
85Labeling School WeeksStarting with cell B3, type numbers 1 through 18 going across row 3 (horizontal).Numbers 1 through 18 represent the number of the school week.You will end with week 18 in cell S3.
86Labeling DatesNote: You may choose to enter the date of that school week across row 2 to easily identify the school week.
87Entering Benchmarks (3rd Grade ORF) In cell B4, type 92. This is your fall benchmark.In cell S4, type 110. This is your winter benchmark.
88Entering Student Data (Sample) Enter the following numbers, going across row 5, under corresponding week numbers.Week 1 – 74Week 3 – 85Week 4 – 89Week 5 – 69Week 6 – 85Week 7 – 96Week 8 – 90Week 9 – 84Week 10 – 106Week 11 – 94Week 15 – 100
89*CAUTION*If a student was not assessed during a certain week, what do you put in that cell?Why?
90Graphing the Data Highlight cells A4 and A5 through S4 and S5 Follow Excel 2003 or Excel 2007 directions from here
91Graphing the Data Excel 2003 Excel 2007 Across the top of your worksheet, click on “Insert”In that drop-down menu, click on “Chart”Excel 2007Click InsertFind the icon for LineClick the arrow below Line
92Graphing the Data Excel 2003 Excel 2007 A Chart Wizard window will appearExcel 20076 graphics appear
93Graphing the Data Excel 2003 Excel 2007 Choose “Line” Choose “Line with markers…”Excel 2007Choose “Line with markers”
94Graphing the Data Excel 2003 Excel 2007 “Data Range” tab “Columns” Your graph appears
95Graphing the Data Excel 2003 Excel 2007 “Chart Title” “School Week” X Axis“WPM’ Y AxisExcel 2007Change your labels by right clicking on the graph
96Graphing the Data Excel 2003 Excel 2007 Choose where you want your graphExcel 2007Your graph was automatically put into your data spreadsheet
97Graphing the Trendline Excel 2003Right click on any of the student data pointsExcel 2007
98Graphing the Trendline Excel 2003Choose “Linear”Excel 2007
99Graphing the Trendline Excel 2003Choose “Custom” and check box next to “Display equation on chart”Excel 2007
100Graphing the Trendline Clicking on the equation highlights a box around itClicking on the box allows you to move it to a place where you can see it better
101Graphing the Trendline You can repeat the same procedure to have a trendline for the benchmark data pointsSuggestion: label the trendline Expected ROIMove this equation under the first
103Challenge! What was the first equation? What is the slope of that equation?What was the second equation?Describe the achievement gap at the end of the school year.
104Programming Excel Second Semester Calculating Needed RoICalculating Benchmark RoICalculating Student’s Actual RoI
105Calculating Needed RoI In cell T3, type Needed RoIClick on cell T5In the fx line (at top of sheet) type this formula =((S4-B5)/18)Then hit enterYour result is _____ ?This formula simply subtracts the student’s actual middle of year (MOY) benchmark from the expected end of year (EOY) benchmark, then dividing by 18 for the first 18 weeks (1st semester).2
106Calculating Benchmark RoI In cell U3, type Benchmark RoIClick on cell U4In the fx line (at top of sheet) type this formula =SLOPE(B4:S4,B3:S3)Then hit enterYour result should read: ____?This formula considers 18 weeks of benchmark data and provides an average growth or change per week.
107Calculating Student Actual RoI Click on cell U5In the fx line (at top of sheet) type this formula =SLOPE(B5:S5,B3:S3)Then hit enterYour result should read: 1.89This formula considers 18 weeks of student data and provides an average growth or change per week.
108Assuming Linear Growth… Why Graph only 18 Weeks at a Time?Assuming Linear Growth……Finding Curve-linear Growth
111McCrea, 2010 Looked at Rate of Improvement in small 2nd grade sample Found differences in RoI when computed for fall and spring:Ave RoI for fall: WCPMAve RoI for spring: 1.21 WCPM
112Ardoin & Christ, 2008 Slope for benchmarks (3x per year) More growth from fall to winter than winter to spring
113Christ, Yeo, & Silberglitt, in press Growth across benchmarks (3X per year)More growth from fall to winter than winter to springDisaggregated special education population
114Graney, Missall, & Martinez, 2009 Growth across benchmarks (3X per year)More growth from winter to spring than fall to winter with R-CBM.
115Fien, Park, Smith, & Baker, 2010Investigated relationship b/w NWF gains and ORF/ComprehensionFound greater NWF gains in fall than in spring.
116DIBELS (6th) ORF Change in Criteria Fall to WinterWinter to Spring2nd24223rd15184th135th1196th5
117AIMSweb Norms 1st 18 31 2nd 25 17 3rd 22 15 4th 16 13 5th 6th 12 Based on 50th PercentileFall to WinterWinter to Spring1st18312nd25173rd22154th16135th6th12
118Speculation as to why Differences in RoI within the Year Relax instruction after high stakes testing in March/April; a PSSA effect.Depressed BOY benchmark scores due to summer break; a rebound effect (Clemens).Instructional variables could explain differences in Graney (2009) and Ardoin (2008) & Christ (in press) results (Silberglitt).Variability within progress monitoring probes (Ardoin & Christ, 2008) (Lent).
119within a Problem-Solving Model ROI as a Decision Toolwithin a Problem-Solving Model
120Steps Gather the data Ground the data & set goals Interpret the data Figure out how to fit Best Practice into Public Education
124Step 2: Ground the Data1) To what will we compare our student growth data?2) How will we set goals?
125Multiple Ways to Look at Growth Needed GrowthExpected Growth & Percent of Expected GrowthFuchs et. al. (1993) Table of Realistic and Ambitious GrowthGrowth Toward Individual Goal**Best Practices in Setting Progress Monitoring Goals for Academic Skill Improvement (Shapiro, 2008)
126Needed GrowthDifference between student’s BOY (or MOY) score and benchmark score at MOY (or EOY).Example: MOY ORF = 10, EOY benchmark is 40, 18 weeks of instruction (40-10/18=1.67). Student must gain 1.67 wcpm per week to make EOY benchmark.
127Expected Growth Difference between two benchmarks. Example: MOY benchmark is 20, EOY benchmark is 40, expected growth (40-20)/18 weeks of instruction = 1.11 wcpm per week.
128Looking at Percent of Expected Growth Tier ITier IITier IIIGreater than 150%Between 110% & 150%Possible LDBetween 95% & 110%Likely LDBetween 80% & 95%May Need MoreBelow 80%Needs MoreTigard-Tualatin School District (www.ttsd.k12.or.us)
131If Local Criteria are Not an Option Use norms that accompany the measure (DIBELS, AIMSweb, etc.).Use national norms.
132Making Decisions: Best Practice Research has yet to establish a blue print for ‘grounding’ student RoI data.At this point, teams should consider multiple comparisons when planning and making decisions.
133Making Decisions: Lessons From the Field When tracking on grade level, consider an RoI that is 100% of expected growth as a minimum requirement, consider an RoI that is at or above the needed as optimal.So, 100% of expected and on par with needed become the limits of the range within a student should be achieving.
139Determining Instructional Level Independent/Instructional/FrustrationalInstructional often b/w 40th or 50th percentile and 25th percentile.Frustrational level below the 25th percentile.AIMSweb: Survey Level Assessment (SLA).
140Setting Goals off of Grade Level 100% of expected growth not enough.Needed growth only gets to instructional level benchmark, not grade level.Risk of not being ambitious enough.Plenty of ideas, but limited research regarding Best Practice in goal setting off of grade level.
141Possible Solution (A)Weekly probe at instructional level and compare to expected and needed growth rates at instructional level.Ambitious goal: 200% of expected RoI
143Possible Solution (B)Weekly probe at instructional level for sensitive indicator of growth.Monthly probes (give 3, not just 1) at grade level to compute RoI.Goal based on grade level growth (more than 100% of expected).
145What do we do when we do not get the growth we want? When to make a change in instruction and intervention?When to consider SLD?
146When to make a change in instruction and intervention? Enough data points (6 to 10)?Less than 100% of expected growth.Not on track to make benchmark (needed growth).Not on track to reach individual goal.
147When to consider SLD? Continued inadequate response despite: Fidelity with Tier I instruction and Tier II/III intervention.Multiple attempts at intervention.Individualized Problem-Solving approach.Evidence of dual discrepancy…
158Step 4: Figure out how to fit Best Practice into Public Education
159Things to Consider Who is At-Risk and needs progress monitoring? Who will collect, score, enter the data?Who will monitor student growth, when, and how often?What changes should be made to instruction & intervention?What about monitoring off of grade level?
160Who is At-Risk and needs progress monitoring? Below level on universal screeningEntering 4th Grade ExampleDORF (110)ISIP TRWM (55)4Sight (1235)PSSA (1235)Student A1155812551232Student B854812161126Student C723510561048
161Who will collect, score, and enter the data? Using MBSP for math, teachers can administer probes to whole class.DORF probes must be administered one-on-one, and creativity pays off (train and use art, music, library, etc. specialists).Schedule for progress monitoring math and reading every-other week.
163Who will monitor student growth, when, and how often? Best Practices in Data-Analysis Teaming (Kovaleski & Pedersen, 2008)Chambersburg Area School District Elementary Response to Intervention Manual (McCrea et. al., 2008)Derry Township School District Response to Intervention Model (http://www.hershey.k12.pa.us/ /lib/ /_files/Microsoft_Word_-_Response_to_Intervention_Overview_of_Hershey_Elementary_Model.pdf)
164What changes should be made to instruction & intervention? Ensure treatment fidelity!!!!!!!!Increase instructional time (active and engaged)Decrease group sizeGather additional, diagnostic, informationChange the intervention
165Final Exam…Student Data: 27, 29, 26, 34, 27, 32, 39, 45, 43, 49, 51, --, --, 56, 51, 52, --, 57.Benchmark Data: BOY = 40, MOY = 68.What is student’s RoI?How does RoI compare to expected and needed RoIs?What steps would your team take next?What if Benchmarks were 68 and 90 instead?
166The RoI Web Site http://sites.google.com/site/rateofimprovement/ Download powerpoints, handouts, Excel graphs, charts, articles, etc.Caitlin Flinn BennyhoffAndy McCreaMatt Ferchalk