Presentation is loading. Please wait.

Presentation is loading. Please wait.

Universal Screening of Academics and Behavior in an RTI Framework A collaboration of Vanderbilt University and the University of Kansas Funded by U.S.

Similar presentations


Presentation on theme: "Universal Screening of Academics and Behavior in an RTI Framework A collaboration of Vanderbilt University and the University of Kansas Funded by U.S."— Presentation transcript:

1 Universal Screening of Academics and Behavior in an RTI Framework A collaboration of Vanderbilt University and the University of Kansas Funded by U.S. Department of Education. Office of Special Education Programs, Judy Shanley, Project Officer Award No. H324U Daryl Mellard April 1-2, 2008 Virginia’s RTI Institute Fredericksburg, VA

2 Acknowledgements from previous presentations Marcia Invernizzi, U. of Virginia, November 2007 presentation at Roanoke Hugh Catts, U. of Kansas, April 2006 presentation at NRCLD National SEA conference on RTI, KCMO (available at

3 A Little Overview General principles about screening Early reading screening Behavioral screening

4 Screening Component in an RTI Framework 1. Academic and behavioral prediction 2. Measures that are quick, low cost, repeatable, critical (predictive) skills, minimal administration training 3. Question: Student at-risk? 4. Affirmative answer: More attention (assessment/ intervention) to class or student 5. Criteria: Criterion benchmark or normative

5 Screening Accuracy Three influences: Base rate. Diagnosticity. Values of those setting the cutoff (criterion) score (i.e., resource priorities).

6 Constructing Screening Measures Match the curricular demands facing students Proximity of the screening test to the criterion performance (Closer should yield higher accuracy.) Should include multiple, related indicators

7 Screening Accuracy Particular attention is given to the accuracy of screening instruments Particular attention is given to the accuracy of screening instruments Errors in identification can be costly Errors in identification can be costly - over identification - over identification - under identification - under identification Accuracy typically quantified within a clinical decision making model Accuracy typically quantified within a clinical decision making model

8 Clinical Decision Making Model At riskNot at risk Normal RD Outcome True Positive a False Negative b True Negative d False Positive c Screen Sensitivity a / a + b Specificity d / c + d Positive Predictive Power a / a + c Negative Predictive Power d / b + d

9 Accuracy of screening is determined by … How well your instrument separates those who eventually will have a problem from those who will not How well your instrument separates those who eventually will have a problem from those who will not What you choose as a cut-off score What you choose as a cut-off score

10 The Ultimate Screen TP 100 FN 0 FP 0 TN 100

11 More Typical Screen TP 80 FN 20 FP 20 TN 80

12 More Typical Screen TP 90 FN 5 FP 35 TN 70

13 ROC Curve

14 What to Measure? What is the criterion? What is the criterion? Reading comprehension involves a mixture of complex abilities Reading comprehension involves a mixture of complex abilities Role of each changes over time Role of each changes over time

15 Part 2:Early Reading Screening Predictive indicators Predictive indicators Criterion measures Criterion measures Remember: Time interval is a big influence Remember: Time interval is a big influence

16 What to Measure? Variables related to early reading Variables related to early reading - letter knowledge - phonological awareness - rapid naming - vocabulary and grammar - reading itself (non-word or word reading) - reading itself (non-word or word reading)

17 What to Measure? Variables related to later reading Variables related to later reading - word reading - oral reading fluency - oral reading fluency - vocabulary and grammar - vocabulary and grammar - text comprehension - text comprehension

18 Reading Screening Measures Hugh Catts, April 2006,  Texas Primary Reading Inventory (Foorman et al., 1998).  Dynamic assessment model (O’Connor & Jenkins, 1999).  Catts, Fey, Zhang, & Tomblin (2001).  Dynamic Indicators of Basic Early Literacy Skills (DIBELS).  Phonological Awareness Literacy Screening (Invernizzi, Juel, Swank & Meier, 1997).  CBM tools.

19 Early Screening Tools Comprehensive Test of Phonological Processing Comprehensive Test of Phonological Processing Test of Phonological Awareness Test of Phonological Awareness Test of Early Reading Ability Test of Early Reading Ability All correlated with reading outcomes (moderate range), but little data on sensitivity and specificity All correlated with reading outcomes (moderate range), but little data on sensitivity and specificity

20 Texas Primary Reading Inventory (Foorman et al., Designed to be used by teachers to identify children at risk for RD and to further evaluate their strengths and weaknesses in reading-related skills Designed to be used by teachers to identify children at risk for RD and to further evaluate their strengths and weaknesses in reading-related skills 5 screens for K-2 nd grade 5 screens for K-2 nd grade Designed to hold false negatives to a minimum Designed to hold false negatives to a minimum Includes an inventory of secondary measures to help rule out false positives Includes an inventory of secondary measures to help rule out false positives

21 TPRI (1998) K (Dec) predicting end of 1st At riskNot at risk Normal RD Outcome Screen (shorten version) Sensitivity 94.8% Specificity 55.9% Positive Predictive Power 39.1% Negative Predictive Power 97.3% Base rate 23% Risk rate 55.8%

22 Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Standardized and readily available Standardized and readily available Developed to monitor progress and inform instruction Developed to monitor progress and inform instruction

23 DIBELS K (Fall) predicting end of 1st At riskNot at risk Normal RD Outcome Screen ( Initial sound fluency, Letter name fluency) Sensitivity 82.5% Specificity 56.7% Positive Predictive Power 47.9% Negative Predictive Power 87.0% Base rate 32.5% Risk rate 56.0%

24 CBM Tools Letter-Name Fluency Letter-Name Fluency Letter-Sound Fluency Letter-Sound Fluency Initial-Sound Fluency Initial-Sound Fluency Phoneme Segmentation Fluency Phoneme Segmentation Fluency Nonword Reading Fluency Nonword Reading Fluency Oral Reading Fluency Oral Reading Fluency Oral Retell Fluency Oral Retell Fluency Maze Fluency Maze Fluency

25 CBM Tools Assessments given 3 or more times a year to evaluate growth in reading (meeting benchmarks) Assessments given 3 or more times a year to evaluate growth in reading (meeting benchmarks) Each can be considered a screening opportunity Each can be considered a screening opportunity

26 O’Connor & Jenkins (1999) Large battery of preliteracy skills Large battery of preliteracy skills - rapid letter naming (# of letters named from random list in 1 min) random list in 1 min) - syllable deletion (say “baseball” without “ball”) - syllable deletion (say “baseball” without “ball”) - segmenting phonemes (tell me how many sounds in “saw”) sounds in “saw”) - phoneme repetition (say “p” “I” “f” ) - phoneme repetition (say “p” “I” “f” ) Chose cut-off scores to maximize sensitivity Chose cut-off scores to maximize sensitivity

27 O’Connor & Jenkins (1999) K ( Nov) predicting April of 1st At riskNot at risk Normal RD Outcome Screen (phonem seg, RLN, deletion) Sensitivity 100% Specificity 89.3% Positive Predictive Power 39.5% Negative Predictive Power 100% Base rate 6.5% Risk rate 16.5%

28 TPRI (1998) 1 st (Oct) predicting end of 1st At riskNot at risk Normal RD Outcome Screen (letter-sound, blending, word reading) Sensitivity 93.3% Specificity 63.5% Positive Predictive Power 38.8% Negative Predictive Power 97.4% Base rate 19.9% Risk rate 47.7%

29 DIBELS Beginning 1 st NWF predicting end 1st ORF At riskNot at risk Normal RD Outcome Sensitivity 71.7% Specificity 76.6% Positive Predictive Power 59.6% Negative Predictive Power 84.8% Risk rate 39.1% Base rate 32.6%

30 Dynamic Assessment May have advantage over static assessment May have advantage over static assessment Measurement of ability over time in order to monitor progress Measurement of ability over time in order to monitor progress Measurement of learners’ potential over the short term Measurement of learners’ potential over the short term Assessor actively intervenes during the course of the assessment with the goal of intentionally inducing changes in the learner's current level of performance. Assessor actively intervenes during the course of the assessment with the goal of intentionally inducing changes in the learner's current level of performance. “Mini-assessment” of response to intervention “Mini-assessment” of response to intervention

31 O’Connor & Jenkins (1999) Oct 1 st predicting April 1 st At riskNot at risk Normal RD Outcome Screen (phoneme seg, RLN, phoneme repetition) Sensitivity 100% Specificity 87.3% Positive Predictive Power 29.7% Negative Predictive Power 100% Base rate 5.1% Risk rate 17.2%

32 O’Connor & Jenkins (1999) Dynamic Assessment Dynamic Assessment - phoneme segmentation - used Elkonin boxes to progressively teach segmentation of a set of test items teach segmentation of a set of test items words words - score based on the number of trials needed to master the task to master the task

33 O’Conner & Jenkins (1999) Oct 1 st predicting April 1 st (dynamic) At riskNot at risk Normal RD Outcome Screen Sensitivity 90.9% Specificity 95.6% Positive Predictive Power 52.6% Negative Predictive Power 99.5% Base rate 5.1% Risk rate 8.8%

34 Compton, Fuchs, Fuchs, & Bryant (2006) Screened in 1 st (Oct) predicting end of 2 nd Screened in 1 st (Oct) predicting end of 2 nd Measures Measures - CTOPP Sound Matching - CTOPP Sound Matching - CTOPP Rapid Digit Naming - CTOPP Rapid Digit Naming - WJPB-R Oral Vocabulary - WJPB-R Oral Vocabulary - Word Identification Fluency (WIF) - Word Identification Fluency (WIF) Initial level, 5-week slope Initial level, 5-week slope

35 Grade 1 Word-Identification Fluency Teacher: Read these words. Time: 1 minute. twoforcomebecauselastfrom......

36 Compton et al. (2006) 1 st (Oct) predicting end of 2 nd At riskNot at risk Normal RD Outcome Screen (includes WIF level & slope – CTA) Sensitivity 94.6% Specificity 91.7% Positive Predictive Power 71.4% Negative Predictive Power 98.7%

37 Beyond First grade Most common screening for Tier 2 has been oral reading fluency (ORF) Most common screening for Tier 2 has been oral reading fluency (ORF) ORF strongly correlated with 3 rd grade state assessments ORF strongly correlated with 3 rd grade state assessments Strong correlations do not necessarily translate into high sensitivity and specificity Strong correlations do not necessarily translate into high sensitivity and specificity Measurement of level and slope may help Measurement of level and slope may help (e.g., dual discrepancy) (e.g., dual discrepancy) Must deal with potential scaling problems Must deal with potential scaling problems

38 What does research tell us about screening? Can identify children at risk for reading problems Can identify children at risk for reading problems Can be done as early as the fall of kindergarten Can be done as early as the fall of kindergarten Need to choose measures carefully Need to choose measures carefully Must match measures to curriculum Must match measures to curriculum - letter naming - letter naming - phonological awareness - phonological awareness - word reading - word reading - text reading - text reading Must not forget about other factors related to comprehension Must not forget about other factors related to comprehension - oral language - oral language

39 What does research tell us about screening? False positive rates are high and efforts need to be in place to limit the cost of over prediction False positive rates are high and efforts need to be in place to limit the cost of over prediction Progress monitoring within a RTI framework may serve this purpose Progress monitoring within a RTI framework may serve this purpose Need to equate forms to scale Need to equate forms to scale Dynamic assessment has potential Dynamic assessment has potential

40 Screening for Possible Reading Risk GradeCBM ProbeCut-off KindergartenLetter Sound Fluency< 10 letters/minute Grade 1Word Identification Fluency < 15 words on list/minute Grade 2Passage Reading Fluency < 15 words in text/minute Grade 3Passage Reading Fluency < 50 words in text/minute Grade 4Maze Fluency< 10 Maze replacements/ 2.5 minutes Grade 5Maze Fluency< 15 Maze replacements/ 2.5 minutes Grade 6Maze Fluency< 20 Maze replacements/ 2.5 minutes Note: These figures may change pending additional RTI research.

41 Tier 1–Primary Prevention: Confirming Risk Status With PM At the end of 5-8 weeks, student risk status is confirmed or disconfirmed. Grade Inadequate Reading Slope Inadequate Math Computation Slope Inadequate Math Concepts and Applications Slope Kindergarten< 1 (LSF)< 0.20 Grade 1< 1.8 (WIF)< 0.25< 0.30 Grade 2< 1 (PRF)< 0.20< 0.30 Grade 3< 0.75 (PRF)< 0.20< 0.50 Grade 4< 0.25 (Maze) < 0.50 Grade 5< 0.25 (Maze) < 0.50 Grade 6< 0.25 (Maze) < 0.50 Note: These figures may change pending additional RTI research.

42 Dr. Marcia Invernizzi University of Virginia Universal Literacy Screening : First Steps Toward Prevention & Intervention

43 Universal Literacy Screening Screen routinely  Fall – Mid-year – Spring Why?  Ensures that students who need additional support do not go too long before receiving additional instruction/intervention.  Helps identify the “point of entry” into the tiers of RtI intervention & the kinds of supports needed.  Monitors student growth over time

44 Phonological Awareness Literacy Screening The state-provided screening tool for Virginia’s EIRI Consists of three instruments:  PALS-PreK (for preschoolers)  PALS-K (for students in kindergarten)  PALS 1-3 (for students in grades 1-3) Measures young children’s knowledge of important literacy fundamentals.

45 45 PALS Instrument Content Areas PALS-PreKPALS-KPALS 1-3 Print and Word AwarenessX Nursery Rhyme AwarenessX Name WritingX Rhyme AwarenessXX Beginning Sound AwarenessXX Alphabet KnowledgeXXX Letter SoundsXXX Concept of WordXX BlendingXX Sound-to-letterXX SpellingXX Word Recognition in isolationXX FluencyX

46 Universal Literacy Screening Purpose #1:  Identification of children in need of further assessment and/or intervention Solution:  PALS class reports available on the PALS website:

47

48 Universal Literacy Screening Purpose #2:  Provision of feedback about how a class is performing so that classroom-based curriculum or instructional issues can be identified as soon as possible. Solution:  PALS class reports available on the PALS website:

49 49

50

51 Universal Literacy Screening Purpose # 3:  Diagnosis of children who may have had a poor testing experience. Solution:  PALS individual student reports available on the PALS website:

52

53 Online Score Entry & Reporting

54 Activities for Teachers PALS Web site Instructional resources

55 Fitzpatrick, J.. (1997). Phonemic Awareness. Cyprus, CA: Creative Teaching Press. (p. 30)

56 Please contact the PALS office if you have any questions or concerns! UVA-PALS

57 References Catts, H.W., Fey, M.E., Zhang, X., & Tomblin, J.B. (2001). Estimating risk for future reading difficulties in kindergarten children: A research-based model and its clinical implications. Language, Speech, and Hearing Services in the Schools, 32, Catts, H.W., Fey, M.E., Zhang, X., & Tomblin, J.B. (2001). Estimating risk for future reading difficulties in kindergarten children: A research-based model and its clinical implications. Language, Speech, and Hearing Services in the Schools, 32, Compton, D.L., Fuchs, D., Fuchs, L.S., & Bryant, J.D. (2006). Selecting at-risk readers in first grade for early intervention: A two-year longitudinal study of decision rules and procedures. Journal of Educational Psychology, 98, Compton, D.L., Fuchs, D., Fuchs, L.S., & Bryant, J.D. (2006). Selecting at-risk readers in first grade for early intervention: A two-year longitudinal study of decision rules and procedures. Journal of Educational Psychology, 98, Foorman, B.R., Fletcher, J.M., Frances, D.J., Carlson, C.D., Chen, D., Mouzaki, A., Schatschneider, C., Wristers, K., & Taylor, R. (1998). Technical Report Texas Primary Reading Inventory Technical (1998 Edition). Houston, TX: Center for Academic and Reading Skills and University of Houston. Foorman, B.R., Fletcher, J.M., Frances, D.J., Carlson, C.D., Chen, D., Mouzaki, A., Schatschneider, C., Wristers, K., & Taylor, R. (1998). Technical Report Texas Primary Reading Inventory Technical (1998 Edition). Houston, TX: Center for Academic and Reading Skills and University of Houston. O’Connor, R.E., & Jenkins, J.R. (1999). Prediction of reading disabilities in kindergarten and first grade. Scientific Studies of Reading, 3, O’Connor, R.E., & Jenkins, J.R. (1999). Prediction of reading disabilities in kindergarten and first grade. Scientific Studies of Reading, 3,

58 Part 3: Behavioral Screening Challenges in screening behavior Challenges in screening behavior Recognize age influences on predictive and criterion data Recognize age influences on predictive and criterion data Alternative procedures Alternative procedures

59 Challenges in Behavioral Screening van Lier, Verhulst & Crijnen (2003) Screening was originally for detecting the presence or absence of highly specific medical conditions that could be detected in a benign presymptomatic stage and for which treatment was available. Disruptive behaviors lack this specificity and unitary underlying conditions; multi-factorial Lack a well-delineated onset affecting valid detection

60 Other Challenges Referral by regular classroom teachers is regarded as the most vulnerable to bias due to differing expectations (Hersh & Walker, 1983; Lloyd et al., 1991) Referral peak for students with behavioral problems occurs in grade 9 (academic problems grades 2 and 3)

61 Social-environmental Domains of Behavioral Influence Multi-factorial influences across developmental periods  Family  Peer group  School and  Neighborhood Incorporate into a multiple gating procedure (e.g., Loeber, 1990; Lochman, 1995)

62 Behavioral Dimensions Internalizing Depression Phobias Social isolation Peer isolation Externalizing ADHD Oppositional defiant disorder Conduct disorder

63 Extra-Ordinary Challenges Challenges Stressful life events Exposure to community violence Maltreatment Poverty Divorce Maternal mental illness Generalization Developmental consistency Gender consistency Social group consistency

64 Family Survey Items (Multiple item composite score) History of maternal mental illness High maternal anxiety Rigid parental perspectives (attitudes, beliefs, values) Frequency of positive parental interactions, esp. with mother Stressful life events Large family size Minimal maternal education level Head of household in unskilled occupations Disadvantaged minority status Single parenthood

65 Academic Predictors Use an percentile index of the number of -- absences school changes failed courses grades repeated discipline referrals Grade x school level data

66 Student Self-Ratings as Predictors The degree to which the student perceives -- School related  Staff cares about the students.  Racial tension in school between staff and students.  Sense of being treated fairly. Peer related  Work hard in school.  Discuss schoolwork/intellectual topics.  Cheat on tests.  Pressure to use drugs.

67 Instruments for Multi-gating (1) Inattention/Overactivity with Aggression (IOWA Conners) 10 items for teacher rating 1. Fidgeting 2. Hums and makes odd noises 3. Excitable, impulsive 4. Inattentive, easily distracted 5. Fails to finish things, short attention span 6. Quarrelsome 7. Acts “smart” 8. Temper outbursts (explosive/unpredictable) 9. Defiant 10. Uncooperative

68 Multi-gating screening (2) Conners Abbreviated Symptom Questionnaire (CASQ) 10 items Inattention Overactivity Impulsivity

69 Results for consideration  1.5 SD on the IOWA Conners were judged as situationally disruptive in the classroom rather than pervasively disruptive  2.0 SD on the IOWA Conners and CASQ more conservative; students in most need of services

70 Systematic Screening for Behavior Disorders Walker and Severson, 1990 Items on externalizing and internalizing factors 1st gate: Classroom teacher nomination of 3 students 2nd gate: Teacher rating of 3 highest on external and internal factors; best for behavior problems 3rd gate: School staff assess on 2 measures of school adjustment with direct observations Critical Events Index: 33 External & internal behavior problems; Least expensive screen

71 Supplemental measures Academic Engaged Time: 2, 20 minute classroom observations; % engaged Peer Social Behavior: 2, 20 minute playground observations; level, quality, and distribution of behavior

72 School Social Behavior Scale (Merrill, 1993) Two broad domains 1. Social competence: interpersonal skills, self-management skills and academic skills 2. Anti-social behavior: hostile-irritable, violation of school rules, disruptions of school activities Teacher or staff student ratings; 65 items Not so great on internal dimension

73 Revised Behavior Problem Checklist (Quay & Peterson, 1987) Teacher rating scale; ages 5 to 18 Lack of representative national norms Spanish translation 89 items; 20 minutes

74 Psychosocial Constraints Severson, H.H., Walker, H.M., Hope-Doolittle, J., Kratochwill, T.R. & Gresham, F.M. (2007). Proactive, early screening to detect behaviorally at- risk students. Journal of School Psychology, 45, Sameroff. A.J., & Rosenblum, K.L. (2006). Annals of New York Academy of Science, 1094, Sameroff, A.J., Peck, S.C., and Eccles. J. S. (2004). Development and psychopathology, 16, Phelan, P., Yu, H.C., & Davidson, A.L. (1994). Navigating the psychosocial pressures of adolescence: The voices and experiences of high school youth. American Educational Research Journal, 31 (2),

75 School-wide Information System Horner and Sugai Web-based software system Recording, entering, ordering, and reporting Office Detention Referrals

76 Thank You On the nrcld.org Daryl Mellard


Download ppt "Universal Screening of Academics and Behavior in an RTI Framework A collaboration of Vanderbilt University and the University of Kansas Funded by U.S."

Similar presentations


Ads by Google