Presentation is loading. Please wait.

Presentation is loading. Please wait.

Aimee R. Holt, PhD Middle Tennessee State University

Similar presentations


Presentation on theme: "Aimee R. Holt, PhD Middle Tennessee State University"— Presentation transcript:

1 Aimee R. Holt, PhD Middle Tennessee State University
Data Analysis within an RtI2 Framework: Linking Assessment to Intervention Aimee R. Holt, PhD Middle Tennessee State University

2 A systematic and data-based method for addressing academic concerns:
What is RTI2? A systematic and data-based method for addressing academic concerns: identifying defining & resolving Brown-Chidsey & Steege (2010) RtI2 is a way of thinking about and doing instruction. It is a mindset focused on finding the instruction that works best for each child.

3 RTI2 is a general education initiative….
Components of RTI2 High-quality instruction Frequent assessment of academic skills Data-based decision making Brown-Chidsey & Steege (2010) Components of RTI2 High-quality instruction (evidenced based) Frequent assessment of academic skills Data-based decision making The purpose of assessment throughout the RtI2 process is to facilitate instruction. Within an RtI2 framework assessment FOR learning replaces assessment OF learning.

4 Problem Identification
Problem Solving At each tier within RTI2, a problem solving model is employed to make decisions Problem Identification Analyze the Results of Implementation Determine Next Steps Analyze the Assessment Plan Results Develop an Intervention Plan Define the Problem Develop an Assessment Plan Implement Plan Progress Monitor Plan Evaluation Problem Analysis

5 What would Assessment at Tier I look like?

6 Universal Screeners LEAs are required to:
Administer a nationally normed, skills-based universal screener to students at their grade level LEAs are required to: Administer a nationally normed, skills-based universal screener to students at their grade level The screener selected must be Skill-based – because it will be used to determine whether students demonstrate the skills necessary to achieve grade level standards. Universal screening data can indicate which students need additional instruction. They are NOT diagnostic. They are a type of formative assessment rather than summative. While the primary goal of summative assessment is to determine how much has been learned, formative assessment data can be used to identify specific skills that need to be taught and how best to teach them. Tier 1 instruction should be effective for about 80% of the students. When fewer than 80% of the students are successful with Tier 1, core instruction needs to be IMPROVED

7 For K-8, Universal Screeners should be administered 3X per year
In grades 9-12, there are multiple sources of data that can be reviewed, such as: EXPLORE, PLAN and ACT; Tennessee Comprehensive Assessment Program (TCAP) which includes Writing (TCAP-WA), End of Course (EOC), 3-8 Achievement and in , Partnership for Assessment of Readiness for College and Careers (PARCC); TVAAS

8 Characteristics of Appropriate Universal Screening Tools
Helps answer questions about efficiency of core program Aligns with curriculum for each grade level Skills mastery aligns with state mandated year-end assessment Ikeda, Neessen, & Witt (2008). Timing the measures increases their standardization and sensitivity to change (Riley-Tillman et al., 2013)

9 General Outcome Measures (GOM’s) Skill Based Measures
3 Types of CBM’s General Outcome Measures (GOM’s) Skill Based Measures Sub-skill Mastery Measures

10 General Outcome Measures
GOMs sample performance across several goals at the same time capstone tasks Ex. Oral reading fluency Can be used for screening (benchmarking), survey & specific level assessment progress monitoring GOMs are used to sample performance across several goals at the same time by using capstone tasks that are complex in the sense that they can only be accomplished successfully by applying a number of contributing skills Ex. Oral reading fluency Can be used for screening (benchmarking), survey & specific level assessment as well as progress monitoring

11 Skills-Based Measures
SBM are similar to GOM’s but can be used when capstone tasks are not available Ex. Math computation Can be used for screening (benchmarking), survey & specific level assessment progress monitoring Skills- based measures are designed to accomplish many of the functions of GOM’s but can be used when capstone tasks are not available Ex. Math computation Can be used for screening (benchmarking), survey & specific level assessment progress monitoring

12 Subskill Mastery Measures
SMMs are very narrow in focus Ex. Names of letters Should not be used for benchmarking (exception… early skills such as Letter Naming Fluency, Letter Sound Fluency, Number Naming Fluency)

13 Example Reading Skills Typically Assessed by Universal Screeners
Grade Areas Typically Assessed by Universal Screeners 6th Oral Reading Fluency ; Reading for understanding 5th Oral Reading Fluency; Reading for understanding 4th Oral Reading Fluency; Reading for understanding 3rd 2nd 1st Letter Naming Fluency (beginning); Phonemic Awareness; Phonics; Word Identification Fluency; Oral Reading Fluency (end) K Letter Naming Fluency; Phonemic Awareness: Early Phonics Skills including Letter Sound Fluency

14 What would Data Analysis at Tier I look like?

15 Making Decisions about Group Data
Review universal screening data to answer the following questions: Is there a class wide problem? Who needs a Tier II intervention? Be sure to examine students at the margin Does anyone need Tier III now?

16 As a guideline, students below the 25th percentile would be considered “at-risk.” Students who exceed grade level expectations may be considered advanced.

17 Who needs a Tier II or Enrichment?
Winter Benchmark for ORF: 90th %- 153; 25th % - 72; Winter Benchmark for Maze: 90th % - 25; 25th % - 9; Instructional level criteria For contextual reading – 93-97% correct For most other academic skills – 85-90% correct ORF Maze 26 /98% 154/100% 154/85% 26 /79% 68/ 95% 09 /94% Example of 3rd Grade Winter Benchmark Data Row 1 needs enrichment Row 2 is sacrificing speed for accuracy – focus should e on increasing accuracy – can be done with instruction in Tier 1 Row 3 is accurate but slow; probably needs Tier 2 intervention focused on fluency Row 4 is slow and inaccurate; possibly needs Tier 2 intervention focused on phonics skills development 68/88% 08 /80%

18 Examining students at the Margins
Winter Benchmark for ORF: 90th %- 153; 25th % - 72; Winter Benchmark for Maze: 90th % - 25; 25th % - 9; Instructional level criteria For contextual reading – 93-97% correct ORF Maze 75/96% 11 /100% 80/100% 10 /97% 73/82% 11/75% Example of 3rd Grade Winter Benchmark Data Row 1: Fluency is just above 25th % but accuracy is good; Probably does not need Tier 2 - work on fluency in Tier 1 Row 2: Although mazes is slow the student is accurate and ORF is well above the 25th % and highly accurate; might not need Tier 2 Row 3: Fluency is just above 25% but accuracy is poor; Probably does need Tier 2 instruction

19 Identifying who needs Tier III
Winter Benchmark for ORF: 25th % - 72; 10th % -44 Winter Benchmark for Maze: 25th % - 9; 10th % - 6 Instructional level criteria For contextual reading – 93-97% correct ORF Maze 46 / 76% 6 / 80% 42 / 83% 5 / 75%

20 Referral to Tier II Decision Tree
Core literacy instruction has been implemented with fidelity ≥80% of student needs are met by core instruction Differentiated instruction has been provided in a small group within core literacy instruction Student has been present for ≥75% of instructional days Student has passed vision and hearing screening Data indicates performance below the 25th% on universal screening of student achievement compared to national norms Additional Assessment data supports universal screening data

21 What do we mean by linking assessment to intervention?

22 Linking Assessment to Interventions….
Research has shown that effective interventions have certain features in common: Correctly targeted to the student’s deficit Appropriate level of challenge (instructional range) Explicit instruction in the skill Frequent opportunities to practice (respond) Provide immediate corrective feedback (e.g., Brown-Chidsey & Steege, 2010; Burns, Riley-Tillman, & VanDerHeyden, 2013; Burns, VanDerHeyden, & Boice, 2008;) An intervention can not be effective if it is not targeting the student’s skill deficit within a skills based hierarchy. Once the target and the level of challenge have been appropriately identified, then interventions should be linked to that data. The intervention identified that include explicit instruction, with frequent opportunities for the student to practice those skills. Immediate corrective feedback will be essential.

23 Academic Instruction in Reading
Both NCLB and IDEA require that instruction in the general education setting cover all 5 areas of reading identified by the National Reading Panel Phonemic Awareness Phonics Fluency Vocabulary Text Comprehension Strategies

24 Linking the 5 skill areas to 3 SLD areas
Basic Word Reading Phonemic Awareness Phonics Reading Fluency Fluency Reading Comprehension Vocabulary Text Comprehension Strategies

25 Phonological Awareness
A metacognitive understanding that words we hear have internal structures based on sound Research on PA has shown that it exerts an independent causal influence on word-level reading. (Berninger & Wagner, 2008) Phoneme – smallest unit of speech The English language has phonemes

26 Phonics Alphabetic principle - Linking phonological (sound) and orthographic (symbol) features of language (Joseph, 2006) Important for learning how to read and spell National Reading Panel –students with explicit AP instruction showed benefits through the 6th grade Phonological awareness is a prerequisite skill

27 Word Reading Skills - (McCormick, 2003)
Word identification: the instance when a reader accesses one or more strategies to aid in reading words (e.g., applying phonic rules or using analogies) Decoding – blending sounds in words or using letters in words to cue the sounds of others in a word (Joseph, 2006) Word recognition: the instant recall of words or reading words by sight; automaticity

28 Fluency “ The ability to read a text quickly, accurately, and with proper expression” (NRP, p.3-5) Most definitions of fluency include an emphasis on prosody – the ability to read with correct expression, intonation and phrasing (Fletcher et al., 2007) National Reading Panel -Good reading fluency skills improved recognition of novel words, expression during reading, accuracy and comprehension When decoding is an automatic process, reading a connected text becomes effortless and therefore requires little conscious attention; thus more cognitive resources are available for higher-order processing of the meaning of the text (Fletcher et al, 2007) Fluent readers have learned to automatically recognize words and to increase reading rate while maintaining accuracy. Fluency provides a bridge between word recognition and comprehension. Fluent readers recognize words and comprehend at the same time.

29 Vocabulary & Text Comprehension Skills
Vocabulary knowledge – including understanding multiple meanings of words; figurative language etc.. Identifying stated details Sequencing events Recognizing cause and effect relationships Differentiating facts from opinions Recognizing main ideas – getting the gist of the passage Making inferences Drawing conclusions Although they can often recall details, many students with RC difficulties struggle with getting the gist (i.e., formulating main ideas) This leads to problems summarizing and/or drawing inferences They are less accurate answering explicit (stated directly in the reading) questions based on their readings than those without RC difficulties and they can have significant difficulties answering implicit (must make inferences to answer) questions These problems may also occur when listening

30 What Would Assessment at Tier II Look Like?

31 So you have identified your “at risk students”- now what?
You will need to conduct Survey Level Assessment (SLA) for these students Survey Level Assessment (SLA) Can be used to: (a) provide information on the difference between prior knowledge and skills deficits to be used to plan instructional interventions & (b) serve as baseline for progress monitoring Because of the potential number of students involved in Tier 2 activities the focus should be on efficiency. Note: Baseline data is a a minimum 3 stable data points. If the baseline data are variable then more data points will need to be collected. Tier 2 should not require in-depth functional analysis, however, some additional data might be necessary. Students needing Tier 3 services will require more in depth functional assessments before they begin their interventions.

32 “What is the CATEGORY of the problem”
Why is it important to conduct Survey Level Assessments before beginning Tier II interventions? The primary question being addressed by the survey level assessment at Tier II is “What is the CATEGORY of the problem” (What is the specific area of academic deficit?) (e.g., Riley-Tillman, Burns, Gibbons, 2013) Evidenced based interventions are only valid for specific purposes with certain groups of student The universal screening data may only tell you that a problem exists, NOT what the category of the intervention needs to be If you mismatch an evidenced based intervention with a problem it was not designed to address, there is NO reason to think it will work.

33 An Example of Survey Level Assessment Using DIBELS
1) Start at student’s grade level 2)Test backwards by grade until the student has reached the “low risk” benchmark for a given skill •Low risk/ established indicates the student has “mastered” that skill Grade CBM Assessed Benchmarked 6th Oral Reading Fluency Fall, Winter, Spring 5th 4th 3rd 2nd 1st Winter, Spring Nonsense Word Fluency Phoneme Segmentation Fluency Letter Naming Fluency Fall K Winter, Spring Letter Naming Fluency Initial Sound Fluency Fall, Winter

34 For example….. In reading
comprehension & fluency = comprehension intervention comprehension low fluency, but decoding = fluency intervention comprehension fluency decoding, but phonemic awareness skills decoding intervention Riley-Tillman et al., (2013) It might be inappropriate to place a 3th grade student who does poorly on MAZE fluency in a comprehension group especially if they are still having decoding problems. In reading: A student with low comprehension but sufficient fluency would likely benefit from a comprehension intervention A student with low comprehension and low fluency, but acceptable decoding, would likely need a fluency intervention A student with low comprehension, low fluency, and low decoding, but sufficient phonemic awareness skills would likely require a decoding intervention

35 Let’s look at Michael a 2nd grade student
At the fall benchmark, he was identified on ORF as being in the some risk range. His score was 30 wcpm Survey level assessment were conducted using: DORF 1st grade – (fluency) DNWF 1st grade – (decoding) DPSF 1st grade – (phonemic awareness) Problem Identification Problem Analysis

36 DIBELS Scores Representing Skills Mastery
Michael’s Scores DIBELS Scores Representing Skills Mastery DORF – 35 wcpm DNWF – 28 scpm DPSF – 38 pcpm Fall Winter Spring DORF > 20 > 40 DNWF > 24 > 50 DPSF > 35 DLNF > 37 ---

37 What next…. You link your assessment data to an intervention that targets the category of skill deficit that was identified You select progress monitoring probe(s) that assess that skill You set the student’s goal for improvement You can use ROI & Gap Analysis Worksheets to help with this

38 What progress monitoring is not…
It is NOT an instructional method or intervention Think of progress monitoring as a template that can be laid over goals and objectives from an assortment of content areas

39 What Would Data Analysis at Tier II Look Like?

40

41 Referral to Tier III Decision Tree
 Tier II intervention(s) have occurred daily for 30 minutes in addition to core instruction Intervention logs attached (3) Fidelity checks completed and attached Implementation integrity has occurred with at least 80% fidelity Student has been present for ≥75% of intervention sessions  Tier II intervention(s) adequately addressed the student’s area of need

42  Tier II intervention was appropriate and research-based
Research based interventions are: □ Explicit □ Systematic □ Standardized □ Peer reviewed □ Reliable/valid □ Able to be replicated Progress monitoring has occurred with at least weekly data points –OR bi-monthly data points  Gap analysis indicates that student’s progress is not sufficient for making adequate growth with current interventions

43 Does a student require Tier III intervention?
Step 1: Need to check to see if the data can be interpreted A minimum of 8-10 data points, if progress monitoring every other week, OR data points, if progress monitoring weekly to make a data-based decision to change to Tier III. Christ (2006) found that formative assessment data can fluctuate especially the greater the age of the student being monitored, therefore a minimum of approximately 8 to 10 data points are necessary to make reliable decisions.

44 Step 2: Examine Rate of Improvement
You can compare the student’s actual ROI to the goal that was established You can use the ROI worksheets Let’s complete one for Michael

45 Completing the ROI Worksheet for Michael
Assessment Used:  DIBELS NWF Student’s score on first probe administered:  28 Student’s score on last probe administered:  37 Fall benchmark expectation:  24 Spring benchmark expectation:  50  Step 1 ____________ - _____________ / _________ = ___________ Spring benchmark expectation Fall benchmark expectation Number of weeks Typical ROI (slope) First we calculate the typical ROI for this skill 50 24 36 0.72

46 37 28 13 0.69 Next we will determine the student’s current ROI

47 1.44 0.72 0.72 1.08 Now take the Typical ROI we calculated in Step 1 which was 0.72 and multiple it X 2 for an aggressive ROI goal or 1.5 for a reasonable ROI goal. Next determine if the student’s ROI which was 0.69 < the Reasonable or Aggressive ROI goal – in this case it is

48 You also can visually analyze the graphed progress monitoring data
Calculate the trend line of the intervention data points and compare it to the aim (goal) line. If the slope of the trend line is less than the slope of the aim line, the student may need to be moved to Tier III. Especially if it appears that given the student’s current ROI that they will not meet year end grade level standards

49 Dual Discrepancy -A student should be deficient in level and have a poor response to evidenced-based interventions (slope) to the degree that he/she is unlikely to meet benchmarks in a reasonable amount of time without intensive instruction to move: between Tier II to Tier III as well as between Tier III and referral for a comprehensive special education evaluation. (e.g., Brown-Chidsey & Steege, 2008; Lichenstien, 2008)

50 What Would Assessment at Tier III Look Like?

51 Specific Level Assessment
Functional analysis of skills Are used to: (a) identify specific skills deficits; (b) students prior knowledge; & (c) serve as baseline for progress monitoring specific level assessments rely primarily on subskill mastery measures. “drill down” to specific deficits Focus on a functional analysis of the learner’s skills Are used to: identify specific skills deficits; (b) provide SP with additional information about a students prior knowledge; & (c) serve as baseline for progress monitoring While survey level assessments rely primarily on GOMs and some skill based measure, specific level assessments rely primarily on subskill mastery measures. Specific level assessments “drill down” to specific deficits (i.e. vowel patterns)

52 Functional Analysis RIOT/ICEL Matrix R- review I – interview
O – observe T - test I – instruction C – curriculum E – environment L- learner Problems in Tier III should be explored through multiple sources of data and possible hypotheses should be generated.

53 Linking Assessment Data to Intervention at Tier III
The learner focus on alterable learner variables identify academic entry level skills The task level of the material the student is expected to master The instruction research-based methods and management strategies used to deliver curriculum Match = Success Instruction Student Task SAY 1st: We conduct a functional analysis of academic skills within an instructional hierarchy to determine what factors may be contributing to the mismatch between actual and desired levels of performance. Prerequisite skills are analyzed to determine if any weak or missing.

54 Targets for Academic Instructional Materials
Instructional level contextual reading – 93-97% correct other academic skills – 85-90% correct Produce larger gains more quickly Gravois, T.A., & Gickling, E.E. (2008). Best practices in instructional assessment. In A. Thomas & J.Grimes (Eds.), Best practices in school psychology (5th ed., pp ). Bethesda, MD: National Association of School Psychologists. Instructional level criteria For contextual reading – 93-97% correct For most other academic skills – 85-90% correct Research has shown that interventions with materials that stay within this range of difficulty produce larger gains more quickly than those that do not.

55 Phonemic Awareness Hierarchy
identifying initial, final & medial sounds in words Alliteration blending individual sounds to make a whole word Blending breaking a whole word into it’s individual parts Segmenting Deleting: saying the new word created by omitting a syllable or individual sound in a word Substituting: changing the initial, final, or medial sound in a word to create a new word Reversing: saying the sounds of a word in reverse order to create a new word Manipulating In order to recommend an appropriately targeted intervention, it is important to identify the level within the hierarchy where the student is in the instructional range. Examples of intervention activities for each of these levels can be found at the Florida Center for Reading Research website. Daly, Chafouleas, & Skinner (2005)

56 Let’s look at Michael again…..
Specific Level Assessment – Phonics: Decoding Skills test Developmental Spelling Analysis Sight words: Graded word list Phonemic Awareness: LAC 3 Problem Analysis Continuing the case we began in the module addressing Tier II, as you recall, Michael was a 2nd grade student referred because he fell below benchmark in the fall on ORF. Survey level assessments determined that he was still having difficulty with phonics/decoding skills but had established phoneme segmenting skills for 3 & 4 phoneme words as measured by DIBELS. He received 13 week of a standard protocol intervention focused on phonics. Although he made gains his ROI was not acceptable and the team decided to refer him to Tier III. Following the problem solving model, his problem has been identified in general terms – his ORF is below grade level and lack of phonics skills is considered to be contributing to his low reading fluency skills, however, now in Tier III problem analysis is going to much deeper to try to establish the functional relation between prior knowledge/skills and his limited ROI

57 Linking specific level assessment data to interventions….
Basing interventions on direct samples of student’s academic skills has been shown to result in larger effect sizes than interventions derived from other data This is also known as a skill by treatment interaction Burns, Codding, Boice & Lukito, (2010) While normative cognitive and neuropsychological assessment measures can be used as part of specific level assessment data collections, how one uses these measures to guide intervention design should be done with the focus on academic skills. For example, lets consider two middle school students with a student difficulties related to math problem solving due. Sam is having difficulties because of limited math vocabulary knowledge related to deficits in comprehension knowledge. Tami’s difficulties with math problem solving are due to her not being able to engage in multiple step processing to reach a solution as well as difficulties with determining essential from nonessential information which are skills related to fluid reasoning or even executive functioning. These two students would certainly needs a different focus during a Tier 3 intervention. The focus of the intervention in both cases should revolve around acquiring specific math skills not on addressing the underlying processing deficits.

58 What Would Data Analysis at Tier III Look Like?

59 Need to look at 3 areas Level Slope Variability

60 Level Central location of data within a phase
often compared to benchmark (goal/aim line) can also look at mean or median for each phase (e.g., Daly III et all., 2010; Hixson et al., 2008; Riley-Tillman & Burns, 2009) Can conduct a Gap Analysis using the worksheet

61 How the central location changes over time
Slope/Trend How the central location changes over time With academic data we are usually looking for an increase in skills Target students ROI can be compared with peer groups ROI or benchmark (e.g., Daly III et all., 2010; Hixson et al., 2008; Riley-Tillman & Burns, 2009)

62 2 approaches for analyzing slope
Calculate ROI and compare to an identified peer group using the ROI worksheet Plot the trend line and compare the aim (goal) line to the slope (trend) line

63 Variability Should be examined both within and between phases
General rule- most of the variability in the data should explained by the trend line 80% of the data points should fall with in 15% of the trend line

64

65 Referral for SLD Evaluation Decision Tree
Tier III Intervention(s) have occurred daily for 60 minutes in addition to core instruction Intervention logs attached (5) Fidelity checks completed and attached Implementation integrity has occurred with at least 80% fidelity Student has been present for ≥75% of intervention sessions Tier III intervention(s) adequately addressed the student’s area of need

66 Referral for SLD Evaluation Decision Tree
Tier III intervention was appropriate and research-based Research based interventions are: □ Explicit □ Systematic □ Standardized □ Peer reviewed □ Reliable/valid □ Able to be replicated Progress monitoring has occurred with at least weekly data points –OR bi-monthly data points at Tier III Gap analysis indicates that student’s progress is not sufficient for making adequate growth with current interventions

67 Referral for SLD Evaluation Decision Tree
The following have preliminarily been ruled out as the primary cause of the student’s lack of response to intervention □ Visual, motor, or hearing disability □ Emotional disturbance □ Cultural factors □ Environmental or economic factors □ Limited English proficiency □ Excessive absenteeism

68 Deciding to refer for SLD evaluation
As part of the teams decision to refer for an SLD evaluation, a Gap Analysis should be conducted Let’s look at how to complete the Gap Analysis worksheet with Michael

69 Current benchmark expectation
Gap Analysis Assessment Used:  2nd ORF Student’s current benchmark performance:  66 Student’s current rate of improvement (ROI):   1.3 Current benchmark expectation: 90 End of year benchmark expectation: Number of weeks left in the school year: 5 Is Gap Significant? ________ / ________ = _________ □ Yes □ No Current benchmark expectation Current performance Current gap Fall Winter Spring 90 66 1.4

70 Conducting a Gap Analysis
Step 2 90 66 24 24 5 4.8 24 1.3 18

71 Additional Consideration

72 SEM Additionally, we cannot ignore issues such as interpreting CBM scores in light of SEM or CI when those scores are used for such as diagnoses and eligibility determinations For more detailed discussion including suggested SEM guidelines for oral reading fluency scores in grades 1-5 see: Christ, T. J. Silberglitt, B. (2007). Estimates of the standard error of measurement for curriculum-based measures of oral reading fluency. School Psychology Review, 36, pp Just as SEM and CI are psychometric issues important for the interpretation of normative achievement and IQ measures, in the future we can expect that we will need to consider these issues with CBM data. Researchers are beginning to explore this issue in more detail. Here is a source for those of you who might be interested in reading more about these issues.

73 Use of Progress Monitoring in Special Education
Because CBM data can be directly tied to skill development necessary to be successful in the curriculum, they possess a higher level of sensitivity, and allows for graphic representation; they allows for development of a higher quality IEP Progress monitoring should continue after the IEP is initiated Exit criteria can be set to determine if early reevaluation can be completed due to student success.

74 Helpful Resources

75 Helpful Resources from NASP

76

77 Additional Helpful Resources
Guilford Press

78

79

80

81


Download ppt "Aimee R. Holt, PhD Middle Tennessee State University"

Similar presentations


Ads by Google