1. Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. PaTTAN RtII Institute June 14, 2010

2. Objectives There needs to be a standardized procedure for calculating RoI There needs to be a standardized procedure for calculating RoI We’re proposing a method using Simple Linear Regression We’re proposing a method using Simple Linear Regression

3. Overview Importance of RoI Importance of RoI RoI Foundations RoI Foundations A Need for Consistency A Need for Consistency Calculating RoI Calculating RoI Individual Student Graphs Individual Student Graphs Programming Excel Programming Excel Decision Making Decision Making Grounding the Data Grounding the Data Interpreting Growth Interpreting Growth Individual Student Student Groups Considerations Considerations Resources Resources

4. Importance of Graphs Vogel, Dickson, & Lehman, 1990 Vogel, Dickson, & Lehman, 1990 Speeches that included visuals, especially in color, improved: Speeches that included visuals, especially in color, improved: Immediate recall by 8.5% Immediate recall by 8.5% Delayed recall (3 days) by 10.1% Delayed recall (3 days) by 10.1%

5. Importance of Graphs “Seeing is believing.” “Seeing is believing.” Useful for communicating large amounts of information quickly Useful for communicating large amounts of information quickly “A picture is worth a thousand words.” “A picture is worth a thousand words.” Transcends language barriers (Karwowski, 2006) Transcends language barriers (Karwowski, 2006) Responsibility for accurate graphical representations of data Responsibility for accurate graphical representations of data

6. Skills Typically Graphed Reading Reading Oral Reading Fluency (ORF) Oral Reading Fluency (ORF) Word Use Fluency (WUF) Word Use Fluency (WUF) Reading Comprehension Reading Comprehension MAZE MAZE Retell Fluency Retell Fluency Early Literacy Skills Early Literacy Skills Initial Sound Fluency (ISF) Initial Sound Fluency (ISF) Letter Naming Fluency (LNF) Letter Naming Fluency (LNF) Letter Sound Fluency (LSF) Letter Sound Fluency (LSF) Phoneme Segmentation Fluency (PSF) Phoneme Segmentation Fluency (PSF) Nonsense Word Fluency (NWF) Nonsense Word Fluency (NWF) Spelling Spelling Written Expression Written Expression Behavior Behavior Math Math Math Computation Math Facts Early Numeracy Oral Counting Missing Number Number Identification Quantity Discrimination

7. Importance of RoI Multi-tiered model Multi-tiered model Progress monitoring Progress monitoring Data for decision-making Data for decision-making Goal setting (Shapiro, 2008) Goal setting (Shapiro, 2008)

8. Importance of RoI Visual inspection of slope Visual inspection of slope Multiple interpretations Multiple interpretations Instructional services Instructional services Need for explicit guidelines Need for explicit guidelines

9. RoI Foundations Deno, 1985 Deno, 1985 Curriculum-based measurement Curriculum-based measurement General outcome measures General outcome measures Short Short Standardized Standardized Repeatable Repeatable Sensitive to change Sensitive to change

10. RoI Foundations Fuchs & Fuchs, 1998 Fuchs & Fuchs, 1998 Hallmark components of Response to Intervention Hallmark components of Response to Intervention Ongoing formative assessment Ongoing formative assessment Identifying non-responding students Identifying non-responding students Treatment fidelity of instruction Treatment fidelity of instruction Dual discrepancy model Dual discrepancy model One standard deviation from typically performing peers in level and rate One standard deviation from typically performing peers in level and rate

11. RoI Foundations Ardoin & Christ, 2008 Ardoin & Christ, 2008 Slope for benchmarks (3x per year) Slope for benchmarks (3x per year) More growth from fall to winter than winter to spring More growth from fall to winter than winter to spring Might be helpful to use RoI for fall to winter Might be helpful to use RoI for fall to winter And a separate RoI for winter to spring And a separate RoI for winter to spring

12. RoI Foundations Fuchs, Fuchs, Walz, & Germann, 1993 Fuchs, Fuchs, Walz, & Germann, 1993 Typical weekly growth rates Typical weekly growth rates Needed growth Needed growth 1.5 to 2.0 times typical slope to close gap 1.5 to 2.0 times typical slope to close gap Example Example Bob is below benchmark on ORF Bob is below benchmark on ORF Typical slope is 1 wcpm per week growth Typical slope is 1 wcpm per week growth Bob would need slope of 1.5 to 2 to close gap in a reasonable amount of time Bob would need slope of 1.5 to 2 to close gap in a reasonable amount of time

13. RoI Foundations Deno, Fuchs, Marston, & Shin, 2001 Deno, Fuchs, Marston, & Shin, 2001 Slope of frequently non-responsive children approximated slope of children already identified as having a specific learning disability Slope of frequently non-responsive children approximated slope of children already identified as having a specific learning disability

14. RoI Definition Algebraic term: Slope of a line Algebraic term: Slope of a line Vertical change over the horizontal change Vertical change over the horizontal change Rise over run Rise over run m = (y 2 - y 1 ) / (x 2 - x 1 ) m = (y 2 - y 1 ) / (x 2 - x 1 ) Describes the steepness of a line (Gall & Gall, 2007) Describes the steepness of a line (Gall & Gall, 2007)

15. RoI Definition Finding a student’s RoI = finding the slope of a line Finding a student’s RoI = finding the slope of a line Using two data points on that line Using two data points on that line Finding the line itself Finding the line itself Linear regression Linear regression Ordinary Least Squares Ordinary Least Squares

16. RoI & Statistics Gall & Gall, 2007 Gall & Gall, 2007 10 data points are a minimum requirement for a reliable trendline 10 data points are a minimum requirement for a reliable trendline How does that affect the frequency of administering progress monitoring probes? How does that affect the frequency of administering progress monitoring probes?

17. Ongoing Research Using RoI for instructional decisions is not a perfect process Using RoI for instructional decisions is not a perfect process Research is currently looking to address sources of error: Research is currently looking to address sources of error: Christ, 2006 – standard error of measurement for slope Christ, 2006 – standard error of measurement for slope Ardoin & Christ, 2009 – passage difficulty and variability Ardoin & Christ, 2009 – passage difficulty and variability Jenkin, Graff, & Miglioretti, 2009 – frequency of progress monitoring Jenkin, Graff, & Miglioretti, 2009 – frequency of progress monitoring

18. Future Considerations Questions yet to be empirically answered Questions yet to be empirically answered What parameters of RoI indicate a lack of RtI? What parameters of RoI indicate a lack of RtI? How does standard error of measurement play into using RoI for instructional decision making? How does standard error of measurement play into using RoI for instructional decision making? How does RoI vary between standard protocol interventions? How does RoI vary between standard protocol interventions? How does this apply to non-English speaking populations? How does this apply to non-English speaking populations?

19. How is RoI Calculated? Which way is best?

20. Multiple Methods for Calculating Growth “Eye ball” Approach “Eye ball” Approach Last point minus First point Approach Last point minus First point Approach Split Middle Approach Split Middle Approach Linear Regression Approach Linear Regression Approach

24. 1.1 Words Per Week

25. RoI Consistency? Eye Ball ??? Last minus First 0.75 Split Middle* 0.50 Linear Regression 1.10

26. RoI Consistency? If we are not all using the same model to compute RoI, we continue to have the same problems as past models, where under one approach a student meets SLD criteria, but under a different approach, the student does not. If we are not all using the same model to compute RoI, we continue to have the same problems as past models, where under one approach a student meets SLD criteria, but under a different approach, the student does not. Hypothetically, if the RoI cut-off was 0.65 or 0.95, different approaches would come to different conclusions on the same student. Hypothetically, if the RoI cut-off was 0.65 or 0.95, different approaches would come to different conclusions on the same student.

27. Technical Adequacy Without a consensus on how to compute RoI, we risk falling short of having technical adequacy within our model. Without a consensus on how to compute RoI, we risk falling short of having technical adequacy within our model.

28. So, Which RoI Method is Best?

29. Literature shows that Linear Regression is Best Practice Student’s daily test scores…were entered into a computer program…The data analysis program generated slopes of improvement for each level using and Ordinary-Least Squares procedure (Hayes, 1973) and the line of best fit. Student’s daily test scores…were entered into a computer program…The data analysis program generated slopes of improvement for each level using and Ordinary-Least Squares procedure (Hayes, 1973) and the line of best fit. This procedure has been demonstrated to represent CBM achievement data validly within individual treatment phases (Marston, 1988; Shinn, Good, & Stein, in press; Stein, 1987). This procedure has been demonstrated to represent CBM achievement data validly within individual treatment phases (Marston, 1988; Shinn, Good, & Stein, in press; Stein, 1987). Shinn, Gleason, & Tindal, 1989

30. Growth (RoI) Research using Linear Regression Christ, T. J. (2006). Short-term estimates of growth using curriculum based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence intervals. School Psychology Review, 35, 128-133. Christ, T. J. (2006). Short-term estimates of growth using curriculum based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence intervals. School Psychology Review, 35, 128-133. Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using curriculum based measurement to establish growth standards for students with learning disabilities. School Psychology Review, 30, 507-524. Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using curriculum based measurement to establish growth standards for students with learning disabilities. School Psychology Review, 30, 507-524. Good, R. H. (1990). Forecasting accuracy of slope estimates for reading curriculum based measurement: Empirical evidence. Behavioral Assessment, 12, 179-193. Good, R. H. (1990). Forecasting accuracy of slope estimates for reading curriculum based measurement: Empirical evidence. Behavioral Assessment, 12, 179-193. Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22, 27-48. Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22, 27-48.

31. Growth (RoI) Research using Linear Regression Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM progress monitoring. Exceptional Children, 75, 151-163. Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM progress monitoring. Exceptional Children, 75, 151-163. Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for curriculum-based measurement. The Journal of Special Education, 23, 223-233. Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for curriculum-based measurement. The Journal of Special Education, 23, 223-233. Shinn, M. R., Good, R. H., & Stein, S. (1989). Summarizing trend in student achievement: A comparison of methods. School Psychology Review, 18, 356-370. Shinn, M. R., Good, R. H., & Stein, S. (1989). Summarizing trend in student achievement: A comparison of methods. School Psychology Review, 18, 356-370.

32. So, Why Are There So Many Other RoI Models? Ease of application Ease of application How many of us want to calculate OLS Linear Regression formulas (or even remember how)? How many of us want to calculate OLS Linear Regression formulas (or even remember how)?

33. An Easy and Applicable Solution

34. Get Out Your Laptops! Open Microsoft Excel I love ROI

35. Graphing RoI For Individual Students Programming Microsoft Excel to Graph Rate of Improvement: Fall to Winter

36. Setting Up Your Spreadsheet In cell A1, type 3rd Grade ORF In cell A1, type 3rd Grade ORF In cell A2, type First Semester In cell A2, type First Semester In cell A3, type School Week In cell A3, type School Week In cell A4, type Benchmark In cell A4, type Benchmark In cell A5, type the Student’s Name (Swiper Example) In cell A5, type the Student’s Name (Swiper Example)

37. Labeling School Weeks Starting with cell B3, type numbers 1 through 18 going across row 3 (horizontal). Starting with cell B3, type numbers 1 through 18 going across row 3 (horizontal). Numbers 1 through 18 represent the number of the school week. Numbers 1 through 18 represent the number of the school week. You will end with week 18 in cell S3. You will end with week 18 in cell S3.

38. Labeling Dates Note: You may choose to enter the date of that school week across row 2 to easily identify the school week. Note: You may choose to enter the date of that school week across row 2 to easily identify the school week.

39. Entering Benchmarks (3rd Grade ORF) In cell B4, type 77. This is your fall benchmark. In cell B4, type 77. This is your fall benchmark. In cell S4, type 92. This is your winter benchmark. In cell S4, type 92. This is your winter benchmark.

40. Entering Student Data (Sample) Enter the following numbers, going across row 5, under corresponding week numbers. Enter the following numbers, going across row 5, under corresponding week numbers. Week 1 – 41 Week 1 – 41 Week 8 – 62 Week 8 – 62 Week 9 – 63 Week 9 – 63 Week 10 – 75 Week 10 – 75 Week 11 – 64 Week 11 – 64 Week 12 – 80 Week 12 – 80 Week 13 – 83 Week 13 – 83 Week 14 – 83 Week 14 – 83 Week 15 – 56 Week 15 – 56 Week 17 – 104 Week 17 – 104 Week 18 – 74 Week 18 – 74

41. *CAUTION* If a student was not assessed during a certain week, leave that cell blank If a student was not assessed during a certain week, leave that cell blank Do not enter a score of Zero (0) it will be calculated into the trendline and interpreted as the student having read zero words correct per minute during that week. Do not enter a score of Zero (0) it will be calculated into the trendline and interpreted as the student having read zero words correct per minute during that week.

42. Graphing the Data Highlight cells A4 and A5 through S4 and S5 Highlight cells A4 and A5 through S4 and S5 Follow Excel 2003 or Excel 2007 directions from here Follow Excel 2003 or Excel 2007 directions from here

43. Graphing the Data Excel 2003 Excel 2003 Across the top of your worksheet, click on “Insert” Across the top of your worksheet, click on “Insert” In that drop-down menu, click on “Chart” In that drop-down menu, click on “Chart” Excel 2007 Excel 2007 Click Insert Find the icon for Line Click the arrow below Line

44. Graphing the Data Excel 2003 Excel 2003 A Chart Wizard window will appear A Chart Wizard window will appear Excel 2007 Excel 2007 6 graphics appear

45. Graphing the Data Excel 2003 Excel 2003 Choose “Line” Choose “Line” Choose “Line with markers…” Choose “Line with markers…” Excel 2007 Excel 2007 Choose “Line with markers”

46. Graphing the Data Excel 2003 Excel 2003 “Data Range” tab “Data Range” tab “Columns” “Columns” Excel 2007 Excel 2007 Your graph appears

47. Graphing the Data Excel 2003 Excel 2003 “Chart Title” “Chart Title” “School Week” X Axis “School Week” X Axis “WPM’ Y Axis “WPM’ Y Axis Excel 2007 Excel 2007 Change your labels by right clicking on the graph

48. Graphing the Data Excel 2003 Excel 2003 Choose where you want your graph Choose where you want your graph Excel 2007 Excel 2007 Your graph was automatically put into your data spreadsheet

49. Graphing the Trendline Excel 2003 Excel 2003 Right click on any of the student data points Right click on any of the student data points Excel 2007 Excel 2007

50. Graphing the Trendline Excel 2003 Excel 2003 Choose “Linear” Choose “Linear” Excel 2007 Excel 2007

51. Graphing the Trendline Excel 2003 Excel 2003 Choose “Custom” and check box next to “Display equation on chart” Choose “Custom” and check box next to “Display equation on chart” Excel 2007 Excel 2007

52. Graphing the Trendline Clicking on the equation highlights a box around it Clicking on the equation highlights a box around it Clicking on the box allows you to move it to a place where you can see it better Clicking on the box allows you to move it to a place where you can see it better

53. Graphing the Trendline You can repeat the same procedure to have a trendline for the benchmark data points You can repeat the same procedure to have a trendline for the benchmark data points Suggestion: label the trendline Expected ROI Suggestion: label the trendline Expected ROI Move this equation under the first Move this equation under the first

54. Individual Student Graph

55. The equation indicates the slope, or rate of improvement. The equation indicates the slope, or rate of improvement. The number, or coefficient, before "x" is the average improvement, which in this case is the average number of words per minute per week gained by the student. The number, or coefficient, before "x" is the average improvement, which in this case is the average number of words per minute per week gained by the student.

56. Individual Student Graph The rate of improvement, or trendline, is calculated using a linear regression, a simple equation of least squares. The rate of improvement, or trendline, is calculated using a linear regression, a simple equation of least squares. To add additional progress monitoring/benchmark scores once you’ve already created a graph, enter additional scores in Row 5 in the corresponding school week. To add additional progress monitoring/benchmark scores once you’ve already created a graph, enter additional scores in Row 5 in the corresponding school week.

57. Individual Student Graph Remember to leave cells blank for the weeks that no score was obtained. Otherwise, the graph will incorporate that score into the set of data points and into the trendline. Remember to leave cells blank for the weeks that no score was obtained. Otherwise, the graph will incorporate that score into the set of data points and into the trendline.

58. Individual Student Graph The slope can change depending on which week (where) you put the benchmark scores on your chart. The slope can change depending on which week (where) you put the benchmark scores on your chart. Enter benchmark scores based on when your school administers their benchmark assessments for the most accurate depiction of expected student progress. Enter benchmark scores based on when your school administers their benchmark assessments for the most accurate depiction of expected student progress.

59. Options for the Graph Resizing Resizing Coloring Coloring Data Labels Data Labels

60. Programming Excel First Semester Calculating Needed RoI Calculating Actual RoI – Benchmark Calculating Actual RoI - Student

61. Calculating Needed RoI In cell T3, type Needed RoI In cell T3, type Needed RoI Click on cell T5 Click on cell T5 In the fx line (at top of sheet) type this formula =((S4-B5)/18) In the fx line (at top of sheet) type this formula =((S4-B5)/18) Then hit enter Then hit enter Your result should read: 2.83333... Your result should read: 2.83333... This formula simply subtracts the student’s actual beginning of year (BOY) benchmark from the expected middle of year (MOY) benchmark, then dividing by 18 for the first 18 weeks (1st semester). This formula simply subtracts the student’s actual beginning of year (BOY) benchmark from the expected middle of year (MOY) benchmark, then dividing by 18 for the first 18 weeks (1st semester).

62. Calculating Actual RoI - Benchmark In cell U3, type Actual RoI In cell U3, type Actual RoI Click on cell U4 Click on cell U4 In the fx line (at top of sheet) type this formula =SLOPE(B4:S4,B3:S3) In the fx line (at top of sheet) type this formula =SLOPE(B4:S4,B3:S3) Then hit enter Then hit enter Your result should read: 0.8825... Your result should read: 0.8825... This formula considers 18 weeks of benchmark data and provides an average growth or change per week. This formula considers 18 weeks of benchmark data and provides an average growth or change per week.

63. Calculating Actual RoI - Student Click on cell U5 Click on cell U5 In the fx line (at top of sheet) type this formula =SLOPE(B5:S5,B3:S3) In the fx line (at top of sheet) type this formula =SLOPE(B5:S5,B3:S3) Then hit enter Then hit enter Your result should read: 2.5137... Your result should read: 2.5137... This formula considers 18 weeks of student data and provides an average growth or change per week. This formula considers 18 weeks of student data and provides an average growth or change per week.

64. Graphing RoI For Individual Students Programming Microsoft Excel to Graph Rate of Improvement: Winter to Spring

65. Setting Up Your Spreadsheet In cell A1, type 3rd Grade ORF In cell A1, type 3rd Grade ORF In cell A2, type Second Semester In cell A2, type Second Semester In cell A3, type School Week In cell A3, type School Week In cell A4, type Benchmark In cell A4, type Benchmark In cell A5, type the Student’s Name (Swiper Example) In cell A5, type the Student’s Name (Swiper Example)

66. Labeling School Weeks Starting with cell B3, type numbers 1 through 18 going across row 3 (horizontal). Starting with cell B3, type numbers 1 through 18 going across row 3 (horizontal). Numbers 1 through 18 represent the number of the school week. Numbers 1 through 18 represent the number of the school week. You will end with week 18 in cell S3. You will end with week 18 in cell S3.

67. Labeling Dates Note: You may choose to enter the date of that school week across row 2 to easily identify the school week. Note: You may choose to enter the date of that school week across row 2 to easily identify the school week.

68. Entering Benchmarks (3rd Grade ORF) In cell B4, type 92. This is your fall benchmark. In cell B4, type 92. This is your fall benchmark. In cell S4, type 110. This is your winter benchmark. In cell S4, type 110. This is your winter benchmark.

69. Entering Student Data (Sample) Enter the following numbers, going across row 5, under corresponding week numbers. Enter the following numbers, going across row 5, under corresponding week numbers. Week 1 – 74 Week 1 – 74 Week 3 – 85 Week 3 – 85 Week 4 – 89 Week 4 – 89 Week 5 – 69 Week 5 – 69 Week 6 – 85 Week 6 – 85 Week 7 – 96 Week 7 – 96 Week 8 – 90 Week 8 – 90 Week 9 – 84 Week 9 – 84 Week 10 – 106 Week 10 – 106 Week 11 – 94 Week 11 – 94 Week 15 – 100 Week 15 – 100

70. *CAUTION* If a student was not assessed during a certain week, leave that cell blank If a student was not assessed during a certain week, leave that cell blank Do not enter a score of Zero (0) it will be calculated into the trendline and interpreted as the student having read zero words correct per minute during that week. Do not enter a score of Zero (0) it will be calculated into the trendline and interpreted as the student having read zero words correct per minute during that week.

71. Graphing the Data Highlight cells A4 and A5 through S4 and S5 Highlight cells A4 and A5 through S4 and S5 Follow Excel 2003 or Excel 2007 directions from here Follow Excel 2003 or Excel 2007 directions from here

72. Graphing the Data Excel 2003 Excel 2003 Across the top of your worksheet, click on “Insert” Across the top of your worksheet, click on “Insert” In that drop-down menu, click on “Chart” In that drop-down menu, click on “Chart” Excel 2007 Excel 2007 Click Insert Find the icon for Line Click the arrow below Line

73. Graphing the Data Excel 2003 Excel 2003 A Chart Wizard window will appear A Chart Wizard window will appear Excel 2007 Excel 2007 6 graphics appear

74. Graphing the Data Excel 2003 Excel 2003 Choose “Line” Choose “Line” Choose “Line with markers…” Choose “Line with markers…” Excel 2007 Excel 2007 Choose “Line with markers”

75. Graphing the Data Excel 2003 Excel 2003 “Data Range” tab “Data Range” tab “Columns” “Columns” Excel 2007 Excel 2007 Your graph appears

76. Graphing the Data Excel 2003 Excel 2003 “Chart Title” “Chart Title” “School Week” X Axis “School Week” X Axis “WPM’ Y Axis “WPM’ Y Axis Excel 2007 Excel 2007 Change your labels by right clicking on the graph

77. Graphing the Data Excel 2003 Excel 2003 Choose where you want your graph Choose where you want your graph Excel 2007 Excel 2007 Your graph was automatically put into your data spreadsheet

78. Graphing the Trendline Excel 2003 Excel 2003 Right click on any of the student data points Right click on any of the student data points Excel 2007 Excel 2007

79. Graphing the Trendline Excel 2003 Excel 2003 Choose “Linear” Choose “Linear” Excel 2007 Excel 2007

80. Graphing the Trendline Excel 2003 Excel 2003 Choose “Custom” and check box next to “Display equation on chart” Choose “Custom” and check box next to “Display equation on chart” Excel 2007 Excel 2007

81. Graphing the Trendline Clicking on the equation highlights a box around it Clicking on the equation highlights a box around it Clicking on the box allows you to move it to a place where you can see it better Clicking on the box allows you to move it to a place where you can see it better

82. Graphing the Trendline You can repeat the same procedure to have a trendline for the benchmark data points You can repeat the same procedure to have a trendline for the benchmark data points Suggestion: label the trendline Expected ROI Suggestion: label the trendline Expected ROI Move this equation under the first Move this equation under the first

83. Individual Student Graph

84. The equation indicates the slope, or rate of improvement. The equation indicates the slope, or rate of improvement. The number, or coefficient, before "x" is the average improvement, which in this case is the average number of words per minute per week gained by the student. The number, or coefficient, before "x" is the average improvement, which in this case is the average number of words per minute per week gained by the student.

85. Individual Student Graph The rate of improvement, or trendline, is calculated using a linear regression, a simple equation of least squares. The rate of improvement, or trendline, is calculated using a linear regression, a simple equation of least squares. To add additional progress monitoring/benchmark scores once you’ve already created a graph, enter additional scores in Row 5 in the corresponding school week. To add additional progress monitoring/benchmark scores once you’ve already created a graph, enter additional scores in Row 5 in the corresponding school week.

86. Individual Student Graph Remember to leave cells blank for the weeks that no score was obtained. Otherwise, the graph will incorporate that score into the set of data points and into the trendline. Remember to leave cells blank for the weeks that no score was obtained. Otherwise, the graph will incorporate that score into the set of data points and into the trendline.

87. Individual Student Graph The slope can change depending on which week (where) you put the benchmark scores on your chart. The slope can change depending on which week (where) you put the benchmark scores on your chart. Enter benchmark scores based on when your school administers their benchmark assessments for the most accurate depiction of expected student progress. Enter benchmark scores based on when your school administers their benchmark assessments for the most accurate depiction of expected student progress.

88. Options for the Graph Resizing Resizing Coloring Coloring Data Labels Data Labels

89. Programming Excel Second Semester Calculating Needed RoI Calculating Actual RoI – Benchmark Calculating Actual RoI - Student

90. Calculating Needed RoI In cell T3, type Needed RoI In cell T3, type Needed RoI Click on cell T5 Click on cell T5 In the fx line (at top of sheet) type this formula =((S4-B5)/18) In the fx line (at top of sheet) type this formula =((S4-B5)/18) Then hit enter Then hit enter Your result should read: 2 Your result should read: 2 This formula simply subtracts the student’s actual middle of year (MOY) benchmark from the expected end of year (EOY) benchmark, then dividing by 18 for the first 18 weeks (1st semester). This formula simply subtracts the student’s actual middle of year (MOY) benchmark from the expected end of year (EOY) benchmark, then dividing by 18 for the first 18 weeks (1st semester).

91. Calculating Actual RoI - Benchmark In cell U3, type Actual RoI In cell U3, type Actual RoI Click on cell U4 Click on cell U4 In the fx line (at top of sheet) type this formula =SLOPE(B4:S4,B3:S3) In the fx line (at top of sheet) type this formula =SLOPE(B4:S4,B3:S3) Then hit enter Then hit enter Your result should read: 1.06 Your result should read: 1.06 This formula considers 18 weeks of benchmark data and provides an average growth or change per week. This formula considers 18 weeks of benchmark data and provides an average growth or change per week.

92. Calculating Actual RoI - Student Click on cell U5 Click on cell U5 In the fx line (at top of sheet) type this formula =SLOPE(B5:S5,B3:S3) In the fx line (at top of sheet) type this formula =SLOPE(B5:S5,B3:S3) Then hit enter Then hit enter Your result should read: 1.89 Your result should read: 1.89 This formula considers 18 weeks of student data and provides an average growth or change per week. This formula considers 18 weeks of student data and provides an average growth or change per week.

93. ROI as a Decision Tool within a Problem-Solving Model

94. Steps 1. Gather the data 2. Ground the data & set goals 3. Interpret the data 4. Figure out how to fit Best Practice into Public Education

95. Step 1: Gather Data Universal Screening Progress Monitoring

96. Common Screenings in PA DIBELS DIBELS AIMSweb AIMSweb MBSP MBSP 4Sight 4Sight PSSA PSSA

97. Validated Progress Monitoring Tools DIBELS DIBELS AIMSweb AIMSweb MBSP MBSP www.studentprogress.org www.studentprogress.org www.studentprogress.org

98. Step 2: Ground the Data 1) To what will we compare our student growth data? 2) How will we set goals?

99. Multiple Ways to Look at Growth Needed Growth Needed Growth Expected Growth & Percent of Expected Growth Expected Growth & Percent of Expected Growth Fuchs et. al. (1993) Table of Realistic and Ambitious Growth Fuchs et. al. (1993) Table of Realistic and Ambitious Growth Growth Toward Individual Goal* Growth Toward Individual Goal* *Best Practices in Setting Progress Monitoring Goals for Academic Skill Improvement (Shapiro, 2008)

100. Needed Growth Difference between student’s BOY (or MOY) score and benchmark score at MOY (or EOY). Difference between student’s BOY (or MOY) score and benchmark score at MOY (or EOY). Example: MOY ORF = 10, EOY benchmark is 40, 18 weeks of instruction (40-10/18=1.67). Student must gain 1.67 wcpm per week to make EOY benchmark. Example: MOY ORF = 10, EOY benchmark is 40, 18 weeks of instruction (40-10/18=1.67). Student must gain 1.67 wcpm per week to make EOY benchmark.

101. Expected Growth Difference between two benchmarks. Difference between two benchmarks. Example: MOY benchmark is 20, EOY benchmark is 40, expected growth (40- 20)/18 weeks of instruction = 1.11 wcpm per week. Example: MOY benchmark is 20, EOY benchmark is 40, expected growth (40- 20)/18 weeks of instruction = 1.11 wcpm per week.

102. Tigard-Tualatin School District (www.ttsd.k12.or.us) Looking at Percent of Expected Growth Tier ITier IITier III Greater than 150% Between 110% & 150% Possible LD Between 95% & 110% Likely LD Between 80% & 95% May Need More Likely LD Below 80%Needs More Likely LD

103. Fuchs, Fuchs, Hamlett, Walz, & Germann (1993) Oral Reading Fluency Adequate Response Table Realistic Growth Ambitious Growth 1 st 2.03.0 2 nd 1.52.0 3 rd 1.01.5 4 th 0.91.1 5 th 0.50.8

104. Fuchs, Fuchs, Hamlett, Walz, & Germann (1993) Digit Fluency Adequate Response Table Realistic Growth Ambitious Growth 1 st 0.30.5 2 nd 0.30.5 3 rd 0.30.5 4 th 0.751.2 5 th 0.751.2

105. From Where Should Benchmarks/Criteria Come? Appears to be a theoretical convergence on use of local criteria (what scores do our students need to have a high probability of proficiency?) when possible. Appears to be a theoretical convergence on use of local criteria (what scores do our students need to have a high probability of proficiency?) when possible.

106. Steps to Develop Local Criteria Not enough time today! Not enough time today! See us in State College in the fall, or See us in State College in the fall, or Check out our website later this summer. Check out our website later this summer.

107. If Local Criteria are Not an Option Use norms that accompany the measure (DIBELS, AIMSweb, etc.). Use norms that accompany the measure (DIBELS, AIMSweb, etc.). Use national norms. Use national norms.

108. Making Decisions: Best Practice Research has yet to establish a blue print for ‘grounding’ student RoI data. Research has yet to establish a blue print for ‘grounding’ student RoI data. At this point, teams should consider multiple comparisons when planning and making decisions. At this point, teams should consider multiple comparisons when planning and making decisions.

109. Making Decisions: Lessons From the Field When tracking on grade level, consider an RoI that is 100% of expected growth as a minimum requirement, consider an RoI that is at or above the needed as optimal. When tracking on grade level, consider an RoI that is 100% of expected growth as a minimum requirement, consider an RoI that is at or above the needed as optimal. So, 100% of expected and on par with needed become the limits of the range within a student should be achieving. So, 100% of expected and on par with needed become the limits of the range within a student should be achieving.

111. Research Support RoI by half year vs. whole year (Curvilinear Growth). RoI by half year vs. whole year (Curvilinear Growth). Expected growth as mediated by initial level. Expected growth as mediated by initial level.

112. Example of Curvilinear Growth BOY to MOY = 1.60 MOY to BOY = 1.19 *BOY to EOY = 1.35

113. Ardoin & Christ, 2008 Slope for benchmarks (3x per year) Slope for benchmarks (3x per year) More growth from fall to winter than winter to spring More growth from fall to winter than winter to spring

114. Christ, Yeo, & Silberglitt, in press Growth across benchmarks (3X per year) Growth across benchmarks (3X per year) More growth from fall to winter than winter to spring More growth from fall to winter than winter to spring Disaggregated special education population Disaggregated special education population

115. Graney, Missall, & Martinez, 2009 Growth across benchmarks (3X per year) Growth across benchmarks (3X per year) More growth from winter to spring than fall to winter with R-CBM. More growth from winter to spring than fall to winter with R-CBM.

116. Fien, Park, Smith, & Baker, 2010 Investigated relationship b/w NWF gains and ORF/Comprehension Investigated relationship b/w NWF gains and ORF/Comprehension Found greater NWF gains in fall than in spring. Found greater NWF gains in fall than in spring.

117. DIBELS ORF Change in Criteria Fall to WinterWinter to Spring 2 nd 2422 3 rd 1518 4 th 13 5 th 119 6 th 115

118. AIMSweb Norms Based on 50 th Percentile Fall to WinterWinter to Spring 1 st 1831 2 nd 2517 3 rd 2215 4 th 1613 5 th 1715 6 th 1312

119. Speculation as to why Differences in RoI within the Year Relax instruction after high stakes testing in March/April; a PSSA effect. Relax instruction after high stakes testing in March/April; a PSSA effect. Depressed BOY benchmark scores due to summer break; a rebound effect (Clemens). Depressed BOY benchmark scores due to summer break; a rebound effect (Clemens). Instructional variables could explain differences in Graney (2009) and Ardoin (2008) & Christ (in press) results (Silberglitt). Instructional variables could explain differences in Graney (2009) and Ardoin (2008) & Christ (in press) results (Silberglitt). Variability within progress monitoring probes (Ardoin & Christ, 2008) (Lent). Variability within progress monitoring probes (Ardoin & Christ, 2008) (Lent).

120. RoI Research Growth Mediated by Level

121. Fien, Park, Smith, & Baker, 2010 Nonsense Word Fluency Nonsense Word Fluency Different growth rates depending on beginning level Different growth rates depending on beginning level

122. Clemens, 2010 Investigated NWF and WIF Investigated NWF and WIF “ NWF slope validity increased as initial skills were lower, but relationships with outcomes similar to WIF” “ NWF slope validity increased as initial skills were lower, but relationships with outcomes similar to WIF”

123. Silberglitt & Hintze, 2007 R-CBM R-CBM Differences in growth rates depending on level Differences in growth rates depending on level Lowest and highest deciles had least amount of growth Lowest and highest deciles had least amount of growth

124. Good et. al., 2010 Growth Rate as Function of Level at BOY (2 nd Grade) 20th40th60th80 th Intensive 0 to 50.110.330.560.98 6 to 150.400.701.051.53 16 to 250.951.431.782.20 Strategic 26 to 341.301.732.062.43 35 to 431.501.832.112.50

125. Conclusions… Appear to be different RoI within the school year. Appear to be different RoI within the school year. Compute RoI goals by half-year (Fall to Winter, Winter to Spring). Compute RoI goals by half-year (Fall to Winter, Winter to Spring). Actual or Expected RoI appears to differ depending on the level at which a student originally scores, which could have goal setting ramifications. Actual or Expected RoI appears to differ depending on the level at which a student originally scores, which could have goal setting ramifications.

126. Step 3: Interpreting Growth

127. What do we do when we do not get the growth we want? When to make a change in instruction and intervention? When to make a change in instruction and intervention? When to consider SLD? When to consider SLD?

128. When to make a change in instruction and intervention? Enough data points (6 to 10)? Enough data points (6 to 10)? Less than 100% of expected growth. Less than 100% of expected growth. Not on track to make benchmark (needed growth). Not on track to make benchmark (needed growth). Not on track to reach individual goal. Not on track to reach individual goal.

129. When to consider SLD? Continued inadequate response despite: Fidelity with Tier I instruction and Tier II/III intervention. Fidelity with Tier I instruction and Tier II/III intervention. Multiple attempts at intervention. Multiple attempts at intervention. Individualized Problem-Solving approach. Individualized Problem-Solving approach.

130. Three Levels of Examples Whole Class Whole Class Small Group Small Group Individual Student Individual Student - Academic Data - Behavior Data

131. Whole Class Example

132. 3 rd Grade Math Whole Class Who’s responding? Who’s responding? Effective math instruction? Effective math instruction? Who needs more? Who needs more? N=19 N=19 4 > 100% growth 4 > 100% growth 15 < 100% growth 15 < 100% growth 9 w/ negative growth 9 w/ negative growth

133. Small Group Example

134. Intervention Group Intervention working for how many? Intervention working for how many? Can we assume fidelity of intervention based on results? Can we assume fidelity of intervention based on results? Who needs more? Who needs more?

135. Individual Kid Example

136. Individual Kid Making growth? Making growth? How much (65% of expected growth). How much (65% of expected growth). Atypical growth across the year (last 3 data points). Atypical growth across the year (last 3 data points). Continue? Make a change? Need more data? Continue? Make a change? Need more data?

137. RoI and Behavior?

139. Step 4: Figure out how to fit Best Practice into Public Education

140. Things to Consider Who is At-Risk and needs progress monitoring? Who is At-Risk and needs progress monitoring? Who will collect, score, enter the data? Who will collect, score, enter the data? Who will monitor student growth, when, and how often? Who will monitor student growth, when, and how often? What changes should be made to instruction & intervention? What changes should be made to instruction & intervention? What about monitoring off of grade level? What about monitoring off of grade level?

141. Who is At-Risk and needs progress monitoring? Below level on universal screening Below level on universal screening Entering 4 th Grade Example DORF (110) ISIP TRWM (55) 4Sight (1235) PSSA (1235) Student A1155812551232 Student B854812161126 Student C723510561048

142. Who will collect, score, and enter the data? Using MBSP for math, teachers can administer probes to whole class. Using MBSP for math, teachers can administer probes to whole class. DORF probes must be administered one- on-one, and creativity pays off (train and use art, music, library, etc. specialists). DORF probes must be administered one- on-one, and creativity pays off (train and use art, music, library, etc. specialists). Schedule for progress monitoring math and reading every-other week. Schedule for progress monitoring math and reading every-other week.

143. Week 1Week 2 ReadingMathReadingMath 1 st XX 2 nd XX 3 rd XX 4 th XX 5 th XX

144. Who will monitor student growth, when, and how often? Best Practices in Data-Analysis Teaming (Kovaleski & Pedersen, 2008) Best Practices in Data-Analysis Teaming (Kovaleski & Pedersen, 2008) Chambersburg Area School District Elementary Response to Intervention Manual (McCrea et. al., 2008) Chambersburg Area School District Elementary Response to Intervention Manual (McCrea et. al., 2008) Derry Township School District Response to Intervention Model (http://www.hershey.k12.pa.us/56039310111408/lib/56039310111408/_files/Microsoft_Word_- _Response_to_Intervention_Overview_of_Hershey_Elementary_Model.pdf) Derry Township School District Response to Intervention Model (http://www.hershey.k12.pa.us/56039310111408/lib/56039310111408/_files/Microsoft_Word_- _Response_to_Intervention_Overview_of_Hershey_Elementary_Model.pdf)

145. What changes should be made to instruction & intervention? Ensure treatment fidelity!!!!!!!! Ensure treatment fidelity!!!!!!!! Increase instructional time (active and engaged) Increase instructional time (active and engaged) Decrease group size Decrease group size Gather additional, diagnostic, information Gather additional, diagnostic, information Change the intervention Change the intervention

146. When Instructional Level is Not the Same as Grade Level Understand needed and expected RoI within broader context: Understand needed and expected RoI within broader context: Needed growth will only get student to next level by next benchmark (as opposed to on level). Needed growth will only get student to next level by next benchmark (as opposed to on level). 100% of expected growth may not be an acceptable minimum (not enough growth b/c level is so low). 100% of expected growth may not be an acceptable minimum (not enough growth b/c level is so low).

147. Grounding RoI When Monitoring Off of Grade Level: Three Options Best Practices in Setting Progress Monitoring Goals for Academic Skill Improvement (Shapiro, 2008). Best Practices in Setting Progress Monitoring Goals for Academic Skill Improvement (Shapiro, 2008). Shinn approach as detailed in AIMSweb training workshop on Progress Monitoring. Shinn approach as detailed in AIMSweb training workshop on Progress Monitoring. Tigard-Tualatin SD Chart: 150% instead of 100% as minimum RoI requirement??? Tigard-Tualatin SD Chart: 150% instead of 100% as minimum RoI requirement???

148. Questions? & Comments!

149. Resources www.interventioncentral.com www.interventioncentral.com www.interventioncentral.com www.aimsweb.com www.aimsweb.com www.aimsweb.com http://dibels.uoregon.edu http://dibels.uoregon.edu http://dibels.uoregon.edu www.nasponline.org www.nasponline.org www.nasponline.org

150. Resources www.fcrr.org www.fcrr.org www.fcrr.org Florida Center for Reading Research http://ies.ed.gov/ncee/wwc// http://ies.ed.gov/ncee/wwc// http://ies.ed.gov/ncee/wwc// What Works Clearinghouse http://www.rti4success.org http://www.rti4success.org http://www.rti4success.org National Center on RtI

151. Flinn & McCrea’s RoI Web Site http://sites.google.com/site/rateofimprove ment/ http://sites.google.com/site/rateofimprove ment/ http://sites.google.com/site/rateofimprove ment/ http://sites.google.com/site/rateofimprove ment/ Download powerpoints, handouts, Excel graphs, charts, articles, etc. Download powerpoints, handouts, Excel graphs, charts, articles, etc. Caitlin Flinn Caitlin Flinn c.s.flinn@iup.edu c.s.flinn@iup.edu c.s.flinn@iup.edu Andrew McCrea Andrew McCrea mccreand@chambersburg.k12.pa.us mccreand@chambersburg.k12.pa.us mccreand@chambersburg.k12.pa.us

152. References Ardoin, S. P., & Christ, T. J. (2009). Curriculum- based measurement of oral reading: Standard errors associated with progress monitoring outcomes from DIBELS, AIMSweb, and an experimental passage set. School Psychology Review, 38(2), 266-283. Ardoin, S. P. & Christ, T. J. (2008). Evaluating curriculum-based measurement slope estimates using triannual universal screenings. School Psychology Review, 37(1), 109-125.

153. References Christ, T. J. (2006). Short-term estimates of growth using curriculum-based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence intervals. School Psychology Review, 35(1), 128-133. Deno, S. L. (1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52, 219-232.

154. References Deno, S. L., Fuchs, L.S., Marston, D., & Shin, J. (2001). Using curriculum-based measurement to establish growth standards for students with learning disabilities. School Psychology Review, 30, 507-524. Flinn, C. S. (2008). Graphing rate of improvement for individual students. InSight, 28(3), 10-12.

155. References Fuchs, L. S., & Fuchs, D. (1998). Treatment validity: A unifying concept for reconceptualizing the identification of learning disabilities. Learning Disabilities Research and Practice, 13, 204-219. Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L., & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22, 27-48.

156. References Gall, M.D., & Gall, J.P. (2007). Educational research: An introduction (8th ed.). New York: Pearson. Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM progress monitoring. Exceptional Children, 75, 151-163.

157. References Karwowski, W. (2006). International encyclopedia of ergonomics and human factors. Boca Raton, FL: Taylor & Francis Group, LLC. Shapiro, E. S. (2008). Best practices in setting progress monitoring goals for academic skill improvement. In A. Thomas and J. Grimes (Eds.), Best practices in school psychology V (Vol. 2, pp. 141-157). Bethesda, MD: National Association of School Psychologists.

158. References Vogel, D. R., Dickson, G. W., & Lehman, J. A. (1990). Persuasion and the role of visual presentation support. The UM/3M study. In M. Antonoff (Ed.), Presentations that persuade. Personal Computing, 14.