Presentation is loading. Please wait.

Presentation is loading. Please wait.

Using Local Norms to Support RtI Practices NASP2010.

Similar presentations


Presentation on theme: "Using Local Norms to Support RtI Practices NASP2010."— Presentation transcript:

1 Using Local Norms to Support RtI Practices NASP2010

2 Abstract The purpose of this presentation is to show how local norms can support RtI practices in a large, urban school district. Topics discussed will include: Rationale and creation of local norms, The potential benefits and limitations of their use How the assessment of relative risk can improve the efficiency of resource allocation in the context of an RtI framework

3 RtI RtI is the practice of (1) providing high quality instruction/intervention MATCHED TO STUDENT NEED and (2) using learning rate over time and level of performance to (3) make important educational decisions. These components of RtI are essential to the development of a successful RtI implementation strategy.¹ RtI is the practice of (1) providing high quality instruction/intervention MATCHED TO STUDENT NEED and (2) using learning rate over time and level of performance to (3) make important educational decisions. These components of RtI are essential to the development of a successful RtI implementation strategy.¹ Matching students to instruction cannot be done when you define this based on students other than the ones in front of you. Matching students to instruction cannot be done when you define this based on students other than the ones in front of you. 1 Batsche, G., Elliott, J., Graden, J., Grimes, J., Kovaleski, J., Prasse, D., Reschly, D., Schrag, J., & Tilly, D. (2005). Response to Intervention: PolicyConsiderations and Implementation. Alexandria, VA: National Association of State Directors of Special Education.

4 RtI Triangle While RtI models typically use triangles to illustrate the ideal distribution of students to resources, reality often looks very different. While RtI models typically use triangles to illustrate the ideal distribution of students to resources, reality often looks very different. It is common in many at-risk schools to see an inversion of where the resources are allocated. It is common in many at-risk schools to see an inversion of where the resources are allocated. It is also common to see a hole or a gap at tier 2. It is also common to see a hole or a gap at tier 2. Often the initial distribution of students into the tiers is based on their proficiency status. Often the initial distribution of students into the tiers is based on their proficiency status. Tier 3_____________________5% Tier 2__________________15% Tier 1_____________80% 50%___________Tier 3 10% ___________Tier 2 40% __________Tier 1

5 A Look at the Status Quo Impact of Lawsuit and Building Coordination Impact of Lawsuit and Building Coordination Results have been an increase in the number of total initial referrals Results have been an increase in the number of total initial referrals Psychologist responsibilities are centered on building coordination and evaluations, leaving less time for pre-referral activities Psychologist responsibilities are centered on building coordination and evaluations, leaving less time for pre-referral activities

6 Impact on School Resources Most students referred (whether or not the referral is appropriate) for an evaluation ultimately require formal testing Most students referred (whether or not the referral is appropriate) for an evaluation ultimately require formal testing Costs of Special Ed Referrals Costs of Special Ed Referrals A full evaluation can be valued as high as $3000 (Mirkin & Potter, 1983) A full evaluation can be valued as high as $3000 (Mirkin & Potter, 1983) Also consider the costs of special educational programming Also consider the costs of special educational programming The shift to RtI will largely impact the role of the school psychologist The shift to RtI will largely impact the role of the school psychologist VanDerHeyden, A., Joseph, C., and Gilbertson, D. (2007)

7 Where Does This Put Us and Where Do We Want to Go A large urban district, with limited financial and instructional resources A large urban district, with limited financial and instructional resources Necessity to identify at-risk students and provide services to improve overall district performance Necessity to identify at-risk students and provide services to improve overall district performance An increasingly inefficient evaluation process for special education An increasingly inefficient evaluation process for special education Limited and inconsistent success in implementing RtI using national norms Limited and inconsistent success in implementing RtI using national norms Using Local Norms may address these issues… Using Local Norms may address these issues…

8 What is the Goal? The students skills have not changed just because we look at their data differently The students skills have not changed just because we look at their data differently But with limited resources we can channel them in ways that can increase the systems capacity to be effective. But with limited resources we can channel them in ways that can increase the systems capacity to be effective. Proficiency is still the destination. Using local data to support RtI practices lets us get their with the car we own, not the jet we wish we had. Proficiency is still the destination. Using local data to support RtI practices lets us get their with the car we own, not the jet we wish we had.

9 Reading First DIBELS DIBELS Training and technical support Training and technical support The OSPS Data System The OSPS Data System The use of available national cut scores to allocate local district resources The use of available national cut scores to allocate local district resources dibels.uoregon.edu © University of Oregon Center on Teaching and Learning. All rights reserved.

10

11 Grade Level Spring Data When using National norm data When using National norm data Tier 3: 12 % Tier 2: 55 % Tier 1: 33 %

12 Creating Local Norms =100*percentrank(A:A,A2,2) =100*percentrank(A:A,A2,2) Better than sliced bread!

13 Grade Level Spring Data When using Local norm data When using Local norm data Tier 3: 9 % Tier 2: 24 % Tier 1: 67 %

14 Determine Your Approach Before attacking the data, identify… Before attacking the data, identify… How many children can be effectively serviced (school/district)? How many children can be effectively serviced (school/district)? Does our core service need to change? Does our core service need to change? What will be the cut-offs for which tier? What will be the cut-offs for which tier? RtI tiers defined by percentiles RtI tiers defined by percentiles

15 Pros of Local Norms Decreases bias in decision making Decreases bias in decision making Increases ability to match instruction to student need Increases ability to match instruction to student need Maintains proficiency expectations Maintains proficiency expectations Illuminates identifiable patterns of performance and changes over time Illuminates identifiable patterns of performance and changes over time Stewart, L., & Kaminski, R. (2002) Stewart, L. & Silberglitt, B. (2008)

16 Cons of Local Norms Can be misinterpreted Can be misinterpreted Does not define acceptable performance Does not define acceptable performance Norms are not diagnostic in isolation Norms are not diagnostic in isolation Must adhere to appropriate testing standards (be careful to ensure integrity of administration) Must adhere to appropriate testing standards (be careful to ensure integrity of administration) Stewart, L., & Kaminski, R. (2002) Stewart, L. & Silberglitt, B. (2008)

17 Not Lowering the Bar Not Lowering the Bar Proficiency status and response are two different questions. Proficiency status and response are two different questions. Proficiency tells us if a student has reached a destination. For WI it is officially assessed in the fall with results unavailable till 3/4 th of the school year is over. Proficiency tells us if a student has reached a destination. For WI it is officially assessed in the fall with results unavailable till 3/4 th of the school year is over. Response to local normative data tells us if a student is on the right track, and continued monitoring will estimate when theyll reach their destination. Response to local normative data tells us if a student is on the right track, and continued monitoring will estimate when theyll reach their destination. A students response to their education is evident by using local data to look at growth. A students response to their education is evident by using local data to look at growth.

18 Further Use of Local Norms Can be used to predict outcomes such as: Ending benchmark risk status Proficiency on state assessments Graduation rate Can use receiver operator characteristic curves to predict binary outcomes

19 Creating Cut Scores for Risk on Binary Outcomes Receiver Operating Characteristic Curve (ROC Curve) Receiver Operating Characteristic Curve (ROC Curve) A graphical plot of sensitivity vs. 1- specificity as the threshold point is varied. A graphical plot of sensitivity vs. 1- specificity as the threshold point is varied.

20 What Does a ROC Curve Do? Plots sensitivity (y-axis) and 1-specificity (x-axis) to determine accuracy of prediction Plots sensitivity (y-axis) and 1-specificity (x-axis) to determine accuracy of prediction In choosing a cut score, factors of sensitivity and specificity need to be understood. In choosing a cut score, factors of sensitivity and specificity need to be understood.

21 Sensitivity Sensitivity is the proportion of true positives that were correctly classified as positive given the threshold or cut score used to establish it. Sensitivity is the proportion of true positives that were correctly classified as positive given the threshold or cut score used to establish it. In terms of sickness it is the proportion of people that actually are sick that were correctly classified as being sick (strep) In terms of sickness it is the proportion of people that actually are sick that were correctly classified as being sick (strep) In proficiency terms, it is the proportion of students that actually are not proficient that were correctly classified as at-risk. In proficiency terms, it is the proportion of students that actually are not proficient that were correctly classified as at-risk.

22 Specificity Specificity is the proportion of true negatives that were correctly classified as negative. Specificity is the proportion of true negatives that were correctly classified as negative. In terms of sickness it is the proportion of people that are healthy that were correctly classified as healthy. In terms of sickness it is the proportion of people that are healthy that were correctly classified as healthy. In terms of proficiency it is the proportion of students that are proficient that were correctly classified as proficient. In terms of proficiency it is the proportion of students that are proficient that were correctly classified as proficient. 1-specificity is the proportion of students that are incorrectly classified as being at-risk when in fact they are proficient. 1-specificity is the proportion of students that are incorrectly classified as being at-risk when in fact they are proficient.

23 The ROC Curve The dashed red line represents test B in terms of diagnostic utility. As the threshold varies (cut scores) the test does no better than random at correct classification The dashed red line represents test B in terms of diagnostic utility. As the threshold varies (cut scores) the test does no better than random at correct classification Test A has more utility and Test A has more utility and

24

25 Area Under the Curve Test Result Variable(s):Winter ORF Score AreaStd. Error Asymptotic Sig. Asymptotic 95% Confidence Interval Lower Bound Upper Bound

26 Coordinates of the Curve Test Result Variable(s):Winter ORF Score (Predicting Spring Percentrank) Positive if Less Than or Equal ToSensitivity1 - Specificity

27 Using ROC Curves to Produce Local Cuts Based on State Test Scores

28 Winter/Spring AUC

29 Winter 1 st Grade ORF

30 Spring 1 st Grade ORF

31 DIBELS Oral Reading Fluency Risk Indicators Local and National Norms

32 Making Data Actionable Use the data to determine: How instruction can be matched to student need Whether or not the core instruction is effective Goals Progress monitoring frequency When the goal will be achieved by their current rate of growth Whether or not interventions are effective

33 Potential benefits to using local Norms within RtI Local data takes into consideration the instructional environment of the studentLocal data takes into consideration the instructional environment of the student Appropriate use of local data could help improve overall instruction by making assessment more meaningfulAppropriate use of local data could help improve overall instruction by making assessment more meaningful Improved hit rateImproved hit rate Creating your own triangle, instead of using someone elses rectangleCreating your own triangle, instead of using someone elses rectangle

34 Potential Outcomes and Uses Expanded role of school psychologists Shift in the types of assessments towards screening, progress monitoring and interventionsShift in the types of assessments towards screening, progress monitoring and interventions This shift will result in more direct involvement in student learningThis shift will result in more direct involvement in student learning Helping schools make the connections between assessment and instructional decision makingHelping schools make the connections between assessment and instructional decision making New roles may develop in the areas of data analysis and could include such things as whole school data collection, management, and interpretationNew roles may develop in the areas of data analysis and could include such things as whole school data collection, management, and interpretation Training other school personnel in the understanding and interpretation of local dataTraining other school personnel in the understanding and interpretation of local data

35 For more information: www2.milwaukee.k12.wi.us/rti www2.milwaukee.k12.wi.us/rti Margaret Peters, Margaret Peters, Kelly Witz, Kelly Witz, Robert Latterman, Robert Latterman, Marc Sanders, Marc Sanders, Steve Smith, Steve Smith, Handouts are available for download on the NASP website Handouts are available for download on the NASP website

36 References Batsche, G., Elliott, J., Graden, J., Grimes, J., Kovaleski, J., Prasse, D., Reschly, D., Schrag, J., & Tilly, D. (2005). Response to Intervention: PolicyConsiderations and Implementation. Alexandria, VA: National Association of State Directors of Special Education. Good, R. H., III, & Kaminski, R. A. (1996). Assessment for instructional decisions: Toward a proactive/ prevention model of decision-making for early literacy skills. School Psychology Quarterly, 11(4), Good, R. H., Wallin, J., Simmons, D. C., Kameenui, E. J., & Kaminski, R. A. (2002). System-wide Percentile Ranks for DIBELS Benchmark Assessment (Technical Report 9). Eugene, OR: University of Oregon. Hintze, J., & Silberglitt, B. (2005). A longitudinal examination of the diagnostic accuracy and predictive validity of R-CBM and high-stakes testing. School Psychology Review, 34(3), Shinn, M. (1998). Advanced applications of Curriculum-Based Measurement. New York, NY US: Guilford Press. Stewart, L., & Kaminski, R. (2002). Best practices in developing local norms for academic problem solving. Best Practices in School Psychology IV (Vol. 1, Vol. 2) (pp ). Washington, DC US: National Association of School Psychologists. Stewart, L., & Silberglitt, B. (2008). Best practices in developing academic local norms. Best Practices in School Psychology V (Vol. 2) (pp ). Washington, DC US: National Association of School Psychologists.. VanDerHeyden, A., Joseph, C., and Gilbertson, D. (2007) A multi-year evaluation of the effects of a Response to Intervention (RtI) model on identification of children for special education. Journal of School Psychology, 45,


Download ppt "Using Local Norms to Support RtI Practices NASP2010."

Similar presentations


Ads by Google