Presentation on theme: "Graphing, Calculating, and Interpreting Rate of Improvement"— Presentation transcript:
1Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P.Andrew E. McCrea, M.S., N.C.S.P.NASP ConventionMarch 3, 2010
2ObjectivesThere needs to be a standardized procedure for calculating RoIWe’re proposing a method using Simple Linear Regression
3Overview Importance of RoI RoI Research A Need for Consistency Calculating RoIIndividual Student GraphsProgramming ExcelDecision MakingGrounding the DataInterpreting GrowthIndividual StudentStudent GroupsConsiderationsResources
4Importance of Graphs Vogel, Dickson, & Lehman, 1990 Speeches that included visuals, especially in color, improved:Immediate recall by 8.5%Delayed recall (3 days) by 10.1%
5Importance of Graphs “Seeing is believing.” Useful for communicating large amounts of information quickly“A picture is worth a thousand words.”Transcends language barriers (Karwowski, 2006)Responsibility for accurate graphical representations of data
7Importance of RoI Multi-tiered model Progress monitoring Data for decision-makingGoal setting (Shapiro, 2008)
8Importance of RoI Visual inspection of slope Multiple interpretations Instructional servicesNeed for explicit guidelines
9RoI Research Deno, 1985 Curriculum-based measurement General outcome measuresShortStandardizedRepeatableSensitive to change
10RoI Research Fuchs & Fuchs, 1998 Hallmark components of Response to InterventionOngoing formative assessmentIdentifying non-responding studentsTreatment fidelity of instructionDual discrepancy modelOne standard deviation from typically performing peers in level and rate
11RoI Research Ardoin & Christ, 2008 Slope for benchmarks (3x per year) More growth from fall to winter than winter to springMight be helpful to use RoI for fall to winterAnd a separate RoI for winter to spring
12RoI Research Fuchs, Fuchs, Walz, & Germann, 1993 Example Typical weekly growth ratesNeeded growth1.5 to 2.0 times typical slope to close gapExampleBob is below benchmark on ORFTypical slope is 1 wcpm per week growthBob would need slope of 1.5 to 2 to close gap in a reasonable amount of time
13RoI Research Deno, Fuchs, Marston, & Shin, 2001 Slope of frequently non-responsive children approximated slope of children already identified as having a specific learning disability
14RoI Research Algebraic term: Slope of a line Vertical change over the horizontal changeRise over runm = (y2 - y1) / (x2 - x1)Describes the steepness of a line (Gall & Gall, 2007)
15RoI Research Finding a student’s RoI = finding the slope of a line Using two data points on that lineFinding the line itselfLinear regressionOrdinary Least Squares
16RoI Research Gall & Gall, 2007 10 data points are a minimum requirement for a reliable trendlineHow does that affect the frequency of administering progress monitoring probes?
17RoI ResearchUsing RoI for instructional decisions is not a perfect processResearch is currently looking to address sources of error:Christ, 2006 – standard error of measurement for slopeArdoin & Christ, 2009 – passage difficulty and variabilityJenkin, Graff, & Miglioretti, 2009 – frequency of progress monitoring
18RoI Research Questions yet to be empirically answered What parameters of RoI indicate a lack of RtI?How does standard error of measurement play into using RoI for instructional decision making?How does RoI vary between standard protocol interventions?How does this apply to non-English speaking populations?
25RoI Consistency? Eye Ball ??? Last minus First 0.75 Split Middle* 0.50 Linear Regression1.10
26RoI Consistency?If we are not all using the same model to compute RoI, we continue to have the same problems as past models, where under one approach a student meets SLD criteria, but under a different approach, the student does not.Hypothetically, if the RoI cut-off was 0.65 or 0.95, different approaches would come to different conclusions on the same student.
27Technical AdequacyWithout a consensus on how to compute RoI, we risk falling short of having technical adequacy within our model.
29Literature shows that Linear Regression is Best Practice Student’s daily test scores…were entered into a computer program…The data analysis program generated slopes of improvement for each level using and Ordinary-Least Squares procedure (Hayes, 1973) and the line of best fit.This procedure has been demonstrated to represent CBM achievement data validly within individual treatment phases (Marston, 1988; Shinn, Good, & Stein, in press; Stein, 1987).Shinn, Gleason, & Tindal, 1989
30Growth (RoI) Research using Linear Regression Christ, T. J. (2006). Short-term estimates of growth using curriculum based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence intervals. School Psychology Review, 35,Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using curriculum based measurement to establish growth standards for students with learning disabilities. School Psychology Review, 30,Good, R. H. (1990). Forecasting accuracy of slope estimates for reading curriculum based measurement: Empirical evidence. Behavioral Assessment, 12,Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22,
31Growth (RoI) Research using Linear Regression Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM progress monitoring. Exceptional Children, 75,Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for curriculum-based measurement. The Journal of Special Education, 23,Shinn, M. R., Good, R. H., & Stein, S. (1989). Summarizing trend in student achievement: A comparison of methods. School Psychology Review, 18,
32So, Why Are There So Many Other RoI Models? Ease of applicationHow many of us want to calculate OLS Linear Regression formulas (or even remember how)?
44*CAUTION*If a student was not assessed during a certain week, leave that cell blankDo not enter a score of Zero (0) it will be calculated into the trendline and interpreted as the student having read zero words correct per minute during that week.
45Creating a Graph Highlight the data in Columns C and D Include cells C2 and D2 throughcells C38 and D38Include anyblank cells
46Creating a Graph Excel 2003/Macs Excel 2007 Click Insert Click Chart Find the icon for LineClick the arrow below Line
47Creating a Graph Excel 2003/Macs Excel 2007 A Chart Wizard window will appearExcel 20076 graphics appear
48Creating a Graph Excel 2003/Macs Excel 2007 Choose Line Choose Line with markersExcel 2007Choose Line with markers
49Creating a Graph Excel 2003/Macs Excel 2007 Data Range tab Columns Your graph appears
50Creating a Graph Excel 2003/Macs Excel 2007 Chart Title School Week (X Axis)WPM (Y Axis)Excel 2007Change your labels by clicking on the graph
51Creating a Graph Excel 2003/Macs Excel 2007 Choose where you want your graphExcel 2007Your graph was automatically put into your data spreadsheet
59Individual Student Graph The equation indicates the slope, or rate of improvement.The number, or coefficient, before "x" is the average improvement, which in this case is the average number of words per minute per week gained by the student.
60Individual Student Graph The rate of improvement, or trendline, is calculated using a linear regression, a simple equation of least squares.To add additional progress monitoring/benchmark scores once you’ve already created a graph, enter additional scores in Column D in the corresponding school week.
61Individual Student Graph Remember to leave cells blank for the weeks that no score was obtained.The graph will incorporate that score into the set of data points and into the trendline.
62Individual Student Graph The slope can change depending on which week (where) you put the benchmark scores on your chart.Enter benchmark scores based on when your school administers their benchmark assessments for the most accurate depiction of expected student progress.
63Options for the GraphResizingColoringData Labels
64To Calculate RoI A Formula Programming ExcelTo Calculate RoIA Formula
65RoI FormulaType RoI in cell B39 below the last week of school
66Calculate Expected Slope Click on cell C39Put your cursor at the top next to the fxType =SLOPE(C3:C38,B3:B38)Hit Enter/Return
67Calculate Actual Slope Click on cell D39Put your cursor at the top next to the fxType =SLOPE(D3:D38,B3:B38)Hit Enter/Return
68within a Problem-Solving Model ROI as a Decision Toolwithin a Problem-Solving Model
69Steps Gather the data Ground the data Interpret the data Figure out how to fit Best Practice into Public Education
73To what will we compare our student growth data? Step 2: Ground the DataTo what will we compare our student growth data?
74Multiple Ways to Look at Growth Needed GrowthExpected Growth & Percent of Expected GrowthFuchs et. al. (1993) Table of Realistic and Ambitious GrowthGrowth Toward Individual Goal**Best Practices in Setting Progress Monitoring Goals for Academic Skill Improvement (Shapiro, 2008)
75Needed GrowthDifference between student’s BOY (or MOY) score and benchmark score at MOY (or EOY).Example: MOY ORF = 10, EOY benchmark is 40, 18 weeks of instruction (40-10/18=1.67). Student must gain 1.67 wcpm per week to make EOY benchmark.
76Expected Growth Difference between two benchmarks. Example: MOY benchmark is 20, EOY benchmark is 40, expected growth (40-20)/18 weeks of instruction = 1.11 wcpm per week.
77Looking at Percent of Expected Growth Tier ITier IITier IIIGreater than 150%Between 110% & 150%Possible LDBetween 95% & 110%Likely LDBetween 80% & 95%May Need MoreBelow 80%Needs MoreTigard-Tualatin School District (www.ttsd.k12.or.us)
80Making Decisions: Best Practice Research has yet to establish a blue print for ‘grounding’ student RoI data.At this point, teams should consider multiple comparisons when planning and making decisions.
81Making Decisions: Lessons From the Field When tracking on grade level, consider an RoI that is 100% of expected growth as a minimum requirement, consider an RoI that is at or above the needed as optimal.So, 100% of expected and on par with needed become the limits of the range within a student should be achieving.
84What do we do when we do not get the growth we want? When to make a change in instruction and intervention?When to consider SLD?
85When to make a change in instruction and intervention? Enough data points (6 to 10)?Less than 100% of expected growth.Not on track to make benchmark (needed growth).Not on track to reach individual goal.
86When to consider SLD? Continued inadequate response despite: Fidelity with Tier I instruction and Tier II/III intervention.Multiple attempts at intervention.Individualized Problem-Solving approach.
87Three Levels of Examples Whole ClassSmall GroupIndividual Student- Academic Data- Behavior Data
96Step 4: Figure out how to fit Best Practice into Public Education
97Things to Consider Who is At-Risk and needs progress monitoring? Who will collect, score, enter the data?Who will monitor student growth, when, and how often?What changes should be made to instruction & intervention?What about monitoring off of grade level?
98Who is At-Risk and needs progress monitoring? Below level on universal screeningEntering 4th Grade ExampleDORF (110)ISIP TRWM (55)4Sight (1235)PSSA (1235)Student A1155812551232Student B854812161126Student C723510561048
99Who will collect, score, and enter the data? Using MBSP for math, teachers can administer probes to whole class.DORF probes must be administered one-on-one, and creativity pays off (train and use art, music, library, etc. specialists).Schedule for progress monitoring math and reading every-other week.
101Who will monitor student growth, when, and how often? Best Practices in Data-Analysis Teaming (Kovaleski & Pedersen, 2008)Chambersburg Area School District Elementary Response to Intervention Manual (McCrea et. al., 2008)Derry Township School District Response to Intervention Model (http://www.hershey.k12.pa.us/ /lib/ /_files/Microsoft_Word_-_Response_to_Intervention_Overview_of_Hershey_Elementary_Model.pdf)
102What changes should be made to instruction & intervention? Ensure treatment fidelity!!!!!!!!Increase instructional time (active and engaged)Decrease group sizeGather additional, diagnostic, informationChange the intervention
103When Instructional Level is Not the Same as Grade Level Understand needed and expected RoI within broader context:Needed growth will only get student to next level by next benchmark (as opposed to on level).100% of expected growth may not be an acceptable minimum (not enough growth b/c level is so low).
104Grounding RoI When Monitoring Off of Grade Level: Two Options Best Practices in Setting Progress Monitoring Goals for Academic Skill Improvement (Shapiro, 2008).Tigard-Tualatin SD Chart: 150% instead of 100% as minimum RoI requirement???
107Resources www.fcrr.org Florida Center for Reading Research What Works ClearinghouseNational Center on RtI
108Flinn & McCrea’s RoI Web Site Download powerpoints, handouts, Excel graphs, charts, articles, etc.Caitlin FlinnAndrew McCrea
109ReferencesArdoin, S. P., & Christ, T. J. (2009). Curriculum-based measurement of oral reading: Standard errors associated with progress monitoring outcomes from DIBELS, AIMSweb, and an experimental passage set. School Psychology Review, 38(2),Ardoin, S. P. & Christ, T. J. (2008). Evaluating curriculum-based measurement slope estimates using triannual universal screenings. School Psychology Review, 37(1),
110ReferencesChrist, T. J. (2006). Short-term estimates of growth using curriculum-based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence intervals. School Psychology Review, 35(1),Deno, S. L. (1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52,
111ReferencesDeno, S. L., Fuchs, L.S., Marston, D., & Shin, J. (2001). Using curriculum-based measurement to establish growth standards for students with learning disabilities. School Psychology Review, 30,Flinn, C. S. (2008). Graphing rate of improvement for individual students. InSight, 28(3),
112ReferencesFuchs, L. S., & Fuchs, D. (1998). Treatment validity: A unifying concept for reconceptualizing the identification of learning disabilities. Learning Disabilities Research and Practice, 13,Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L., & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22,
113ReferencesGall, M.D., & Gall, J.P. (2007). Educational research: An introduction (8th ed.). New York: Pearson.Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM progress monitoring. Exceptional Children, 75,
114ReferencesKarwowski, W. (2006). International encyclopedia of ergonomics and human factors. Boca Raton, FL: Taylor & Francis Group, LLC.Shapiro, E. S. (2008). Best practices in setting progress monitoring goals for academic skill improvement. In A. Thomas and J. Grimes (Eds.), Best practices in school psychology V (Vol. 2, pp ). Bethesda, MD: National Association of School Psychologists.
115ReferencesVogel, D. R., Dickson, G. W., & Lehman, J. A. (1990). Persuasion and the role of visual presentation support. The UM/3M study. In M. Antonoff (Ed.), Presentations that persuade. Personal Computing, 14.