Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Illinois Statewide Implementation of the Problem Solving/RtI Initiative NASP 2010 Annual Convention Hyatt Regency, Chicago, IL March 4, 2010 David Bell,

Similar presentations


Presentation on theme: "1 Illinois Statewide Implementation of the Problem Solving/RtI Initiative NASP 2010 Annual Convention Hyatt Regency, Chicago, IL March 4, 2010 David Bell,"— Presentation transcript:

1 1 Illinois Statewide Implementation of the Problem Solving/RtI Initiative NASP 2010 Annual Convention Hyatt Regency, Chicago, IL March 4, 2010 David Bell, St. Xavier University & I-ASPIRE Chicago, Gary L. Cates, Illinois State University & I-ASPIRE Central, Kathryn Cox, Illinois State Board of Education, Ben Ditkowsky, Lincolnwood SD 74 & I-ASPIRE North, Sara Golomb, Doctoral Student, Loyola University Chicago, Mark E. Swerdlik, lllinois State University & I-ASPIRE Central,

2 2 Session Objectives In todays presentation, we will: Provide an overview of Illinois ASPIRE and the major project evaluation components Discuss project evaluation results from three data sources: Self-Assessment of Problem Solving Implementation (SAPSI) Student Outcome Data IHE Checklist (Review of selected educator preparation programs in Illinois) Share some of challenges associated with data collection for project evaluation

3 5-year State Personnel Development Grant from OSEP (now in Year 5) Based on 10+ years of state experience with problem-solving in a 3-tier model Primary Goal: Establish and implement a coordinated, regionalized system of personnel development that will increase the capacity of LEAs to provide early intervening services [with an emphasis on K-3 reading], aligned with the general education curriculum, to at-risk students and students with disabilities, as measured by improved student progress and performance. Illinois ASPIRE Alliance for School-based Problem-solving & Intervention Resources in Education

4 I-ASPIRE Objectives 1.Deliver research-based professional development and technical assistance. 2.Increase the participation of parents in decision- making across district sites. 3.Incorporate professional development content into IHE general and special education preservice curricula. 4.Evaluate the effectiveness of project activities.

5 I-ASPIRE: Regional System for T.A. & Professional Development 4 regional Illinois ASPIRE Centers Illinois ASPIRE - Chicago: Chicago Public Schools Illinois ASPIRE - North: Northern Suburban Special Ed. Illinois ASPIRE - Central: Peoria ROE #48 Illinois ASPIRE - South: Southern Illinois University Edwardsville Collaboratives of LEAs, IHEs, regional providers and parent entities Responsible for: Training to districts and parents in region General technical assistance (T.A.) On-site T.A. to school demonstration sites

6 Illinois T.A. Project Evaluation Coordinated through Loyola University Chicago, Center for School Evaluation, Intervention & Training (CSEIT); aspire.shtmlhttp://www.luc.edu/cseit/i- aspire.shtml Evaluation Framework: 1.If people are trained, do they implement? 2.If people implement, do they do so with fidelity? 3.If people implement with fidelity, do they sustain the practice(s) over time? 4.If people sustain the practice(s), what is the impact on student outcomes (school, group, individual)?

7 I-ASPIRE Evaluation: Key Data Sources Self-Assessment of Problem Solving Implementation (SAPSI): Assesses the degree of implementation of the problem solving process at the building level as self-reported by school sites Fidelity of Implementation Checklist: Designed to assess the degree to which problem solving & RtI processes are implemented as intended; involves a review of products by external reviewer Student Outcome Data: Involves analysis of universal screening, progress monitoring, and state assessment (ISAT) results Parent Survey: Assesses participation (more than satisfaction) in the problem solving process of parents and guardians whose children are receiving Tier 3 interventions IHE Checklist: Designed to assess the amount of RtI content incorporated into IHE general and special education pre-service and graduate curricula

8 SAPSI and Fidelity Of Implementation Checklist Gary L. Cates Illinois State University I-ASPIRE Central 8

9 Self Assessment of Problem- Solving Implementation (SAPSI) School/Administration Focused Problem-Solving Survey 25 questions Completed twice per year Action Planning Document Developed over time and tweaked when necessary 9

10 SAPSI 10

11 SAPSI Outcomes 11

12 SAPSI Outcomes 12

13 SAPSI Outcomes 13

14 SAPSI Outcomes 14

15 SAPSI Outcomes 15

16 SAPSI Outcomes 16

17 SAPSI Outcomes 17

18 Fidelity Checklist External check of fidelity of implementation Completed 1 time per year (Spring) 5 cases from 5 randomly chosen schools Inter-rater Reliability of evaluators >80% Sources: SIP, IPF, Data files, CBM, Training Logs Dichotomous scoring Comments Few Additional Scoring Guidelines for specific Items 18

19 Fidelity Checklist 19

20 Fidelity Outcomes 20

21 Fidelity Outcomes 21

22 Fidelity Outcomes 22

23 Fidelity Outcomes 23

24 Fidelity Outcomes 24

25 Fidelity Outcomes 25

26 Fidelity Outcomes 26

27 Fidelity Outcomes 27

28 Fidelity Outcomes 28

29 Fidelity Outcomes 29

30 Fidelity Outcomes 30

31 Fidelity Outcomes 31

32 Fidelity Outcomes 32

33 Illinois – ASPIRE Northern Region Targeted Program Evaluation Ben Ditkowsky Lincolnwood School District 74 I-ASPIRE North 33

34 Evaluation Question Assumption(s) Successful Implementation of RTI Will Increase Average Overall Achievement and in Particular, ROI for Students Who Receive Tiers 2 and 3 Intervention

35 Facts Increases in Achievement Come from Changes in Curriculum and Instruction, Fidelity of Implementation, Increased Behavior Support, etc

36 Why Use Local Assessments, Such As CBM? State-mandated tests assess outcomes Local assessments allow us to: Measure students earlier than 3 rd grade Monitor progress more frequently than once per year Rely on multiple assessment tools for our information Develop an integrated assessment system with benchmarks for performance, linked to a common outcome

37 Question #1. Do scores on CBM Matter? Source data AIMSWEB

38 How many words did they read in one minute? Sam read 22 Mary read 44 Juan read 65 Dorothy read 94

39 Sam read 22 Mary read 44 Juan read 65 Dorothy read 94

40 Exceed Standards Meets Standards Below Standards Academic Warning

41 in Fall, Dorothy read 94 correct words in a minute She obtained a score of 169 on the state test in the spring

42

43

44

45 Curriculum-Based Measurement is a measure of general reading competence Validity coefficients for R-CBM with the Comprehension subtest of the SAT were.91 as compared with Question Answering.82, Recall.70, Cloze.72 (Fuchs, Fuchs & Maxwell, 1988) Validity coefficients for Text Fluency of Folk Tales with the Iowa Test of Basic Skills Comprehension was.83 (Jenkins, Fuchs, Espin, van den Broek & Deno, 2000) Fluency is causally related to reading comprehension (National Reading Panel -NICHD, 2000)

46 Is fall Curriculum-Based Measurement related to state testing? 62% 38%

47 Big Idea # 1. Scores on CBM are related to results of high-stakes testing

48 Correct = 90% Correct = 81% Correct = 71% Correct = 83% Correct = 88%

49 CBM Is A Reliable Predictor Of ISAT Predictive and Concurrent Validity Grade FallWinterSpring 3r N r N r N r N Note. Data from 8 small to moderate school districts in the Northern Region of Illinois, 2008

50 What Is The Probability Of Meeting Standards On The State Test? Fall R-CBM (Grade 3) Probability of meeting standards on ISAT For a probability of.5 a student would have to read 53 WRC between (51 and 54 WRC) For a probability of.8 a student would have to read 77 WRC between (76 and 80 WRC) Note. Empirical confidence intervals constructed through bootstrapping 100 samples without replacement

51 Cut-Scores For Proficiency Note. Cut scores for grades 3-5 revised based on 2008 ISAT data

52 A Summary of Adequate Progress Using Cut – Scores we can determine the effects of curriculum and instruction on students based on an estimate of initial skill. Kindergarten Measures by Time Students were administered CBM or TELS three points in time (Fall, Winter and Spring). Caveat – Not every district measured the same things or at the same times (i.e., because of local decision-making, the dependent variable is not well controlled) Fall – Letter Naming Fluency Winter – Either Letter Sounds Fluency or Nonsense Word Fluency Spring – Either Letter Sounds Fluency or Nonsense Word Fluency

53 Approximately 70% of students began the year with adequate Print Awareness. But this was sufficient for only about 60% to translate into alphabetic skills. Approximately 12% of students began the year with Questionable Print Awareness. About 40% of these students were able to demonstrate sufficient alphabetic skill in spring Approximately 17% of students began the year with little to no Print Awareness. About 60% demonstrated progress but.. Only 30% demonstrated sufficient alphabetic skill. ALL Schools Effects in K; For Students entering with and Without Print Awareness

54 Proficient 1.2 times more likely to demonstrate PROFICIENCY 1.7 times more likely to demonstrate PROFICIENCY Overall 1.2 times more likely to demonstrate progress A High Fidelity Implementation Site

55 Proficient 1.1 times more likely to demonstrate progress 0.7 times as likely to demonstrate progress Overall 1.1 times more likely to demonstrate progress A Demographically Diverse Site Below Basic 1.3 times more likely to demonstrate progress

56 Proficient Equally likely to demonstrate progress 0.4 times as likely to demonstrate progress Overall 0.9 times more likely to demonstrate progress A New Implementation Site Below Basic 0.4 times as likely to demonstrate progress

57 Quick Summary of SoAP findings for Kindergarten students in the N. Region Print Awareness increases the likelihood of the development of Alphabetic Skill, but it is not enough. Too few students enter kindergarten with sufficient Print Awareness (1 of 3 does not have sufficient print awareness) Students who enter kindergarten with sufficient print awareness are 1.5 times more likely to meet key benchmarks than those with questionable skill; and 2.14 times more likely than those who enter without sufficient skill Only our highest implementation site consistently out performed, the general trend, but even there only 2 of 3 students demonstrated sufficient Progress Caveat – Not every district measured the same things or at the same times (i.e., because of local decision-making, the dependent variable is not well controlled)

58 Growth Over Time; Examination of Cohorts Benchmark Targets

59 Adequate Progress With CBM and TELS On Average IASPIRE North Sites have maintained performance well above expected targets. As new sites enter the project, the overall trends were maintained Ultimately, Schools in Illinois Are Evaluated Based on State Testing Results

60 A High Fidelity Implementation Site 89%97% Progress?

61 82%77% Progress? When Demographics Change

62 A High Poverty Implementation Site 33%27% Progress? Up until 2006, ELL students were assessed with the Illinois Measure of Annual Progress in English

63 74%85% Progress? A Newer Implementation Site

64 Another Newer Implementation Site 41%47% Progress?

65 Conclusion We have seen improvement over time for students. How much improvement we see depends on our focus Ultimately, Increases in Achievement Come from Changes in Curriculum and Instruction, Fidelity of Implementation, Increased Behavior Support, etc

66 Survey of Institutions of Higher Education Mark E. Swerdlik Illinois State University I-ASPIRE Central 66

67 Institutions of Higher Education Checklist- Purpose, Methodology, and Sample Purpose: Review preservice and graduate curricula related to the degree to which they are including school-based problem solving and early intervening services (RtI knowledge and skill set). Methodology: Expert rater reviewed syllabi for relevant courses and interviewed faculty member familiar with preparation program Sample: Five IHEs within the four I-ASPIRE regions. Programs included school psychology, general education, special education and educational leadership/administration 67

68 Sections of IHE Checklist Section 1: Three-Tier Problem Solving and Response to Intervention Section 2: Universal Screening and Problem Identification Section 3: Scientifically Based Reading Instruction in a Three Tier Model Section 4:Scientifically Based Progress Monitoring Tools Section 5: Effective Problem Solving Teams 68

69 Rating Scale for IHE Checklist Rating/Meaning of Score 0 /No evidence that the component is included in the class 1/Component is mentioned in the class 2/Component is mentioned in the class AND there are required readings, assignments, and/or projects for application A higher rating indicated that the curriculum included problem solving, RtI, and early intervening services content. 69

70 Open-Ended Interview How does your program prepare pre-service students to participate in three-tier problem-solving models and Response to Intervention? How does your program prepare pre-service students to participate in universal screening and problem identification as part of this model? How does your program prepare pre-service students to implement scientifically-based reading instruction as part of this model? How does your program prepare pre-service students to implement scientifically-based progress monitoring in a three-tier model? How does your program prepare pre-service students to participate in effective problem-solving teams? Space was left for additional questions that arose from the interview. Only two of the five schools submitting syllabi completed the interview process described. 70

71 Institutions of Higher Education Checklist-Overall little implementation 71

72 Percentage of Items by Section 72

73 School Psychology Highest Percentage of Infusion 73

74 Themes and Subcategories of Interview Field Experience -Students participate in clinical experiences -Students participate in a practicum -Students participate in student teaching. Coursework -Topics are highlighted in class work. -Students must take a special education course dealing with these topics. -A topic running through these interviews is reading strategies. Lack of Implementation/Application -Interviewees indicate a possibility for future courses. 74

75 Conclusions and Implications Low rate of infusion of RtI knowledge and skill set development in preservice preparation programs School psychology graduate preparation programs followed by special education have the highest rates of infusion At some schools this activity has lead to increased communication about the importance of these topics and professional development for program faculties (e.g., Meeting with Public Deans and topic of annual Education Institute) 75

76 Data Collection Challenges David Bell St. Xavier University I-ASPIRE North

77 Data Collection One major component of the grant is focused on retrieving student outcome data (behavior and academic) Some barriers Inconsistency with respect to data that is collected Some districts did not have an appropriate data management system to collect behavior information Additional variables (beyond the grants control) impacted the ability to collect academic screening data

78 SAPSI Collecting reliable information from the participants required revisions to the original document Some participants viewed the document as busy work especially districts that have been implementing the key principles years prior to the grant Multiple administrations seemed overwhelming for participants

79 IHE Tool Required the analysis/interpretation of syllabi and interviews with professors Difficulty with truly understanding the depth of course content just by reviewing syllabi Difficulty establishing the sense of urgency from University professors

80 Additional Tools Fidelity Checklist Challenges of collecting data at a minimum PARENT SURVEY Collecting the information seemed to be a consistent barrier

81 General Information on I-ASPIRE

82 All Evaluation Forms Available aspireresourcesforcoordinator.s html 82

83 83 Questions Thank You for Your Attention!


Download ppt "1 Illinois Statewide Implementation of the Problem Solving/RtI Initiative NASP 2010 Annual Convention Hyatt Regency, Chicago, IL March 4, 2010 David Bell,"

Similar presentations


Ads by Google