Presentation is loading. Please wait.

Presentation is loading. Please wait.

Washington State Using Data to Drive Statewide Improvement Efforts Greg Roberts, Evaluation Research Services (ERS) Leslie Pyper, SPDG Director, Learning.

Similar presentations


Presentation on theme: "Washington State Using Data to Drive Statewide Improvement Efforts Greg Roberts, Evaluation Research Services (ERS) Leslie Pyper, SPDG Director, Learning."— Presentation transcript:

1 Washington State Using Data to Drive Statewide Improvement Efforts Greg Roberts, Evaluation Research Services (ERS) Leslie Pyper, SPDG Director, Learning Improvement Coordinator, Special Education, OSPI SPDG Director’s Webinar November 17,

2 Part 1 (Greg) Evaluation Approach Application of Evaluation Approach Early and Preliminary Findings Part 2 (Leslie) Thinking about Systems Leveraging Evaluation Efforts Creating Systemic Change in WA 2

3 Evaluation Approach Theory-driven evaluation Program theory - what must be done to achieve desired goals, what other important impacts may also be anticipated, and how these goals and impacts would be generated Normative theory and causative theory Treatment refers to the services, materials, and activities thought to be essential to generating desired changes. 3

4 Evaluation Approach Program model as a vehicle for talking about exceedingly complex social phenomena Basis for initially capturing important information Basis for clearly and usefully reporting evaluation activities and findings Means of identifying important research questions Means of identifying program elements and outcomes for measurement Mechanism for Answers for identifying important features of an implementation environment(s) 4

5 Evaluation Approach Causative theory (see figure above) represents empirical and substantive knowledge of the relationships that link a program’s treatments, its implementation processes, and its intended outcomes. Impact theory and impact evaluation are perhaps the most well advertised elements of this framework. 5

6 Washington State Application Program theory/model Evaluation questions Evaluation design Measures, indicators Procedures – Survey – Site Visit – Extant Data 6

7 Program Model 7

8 Evaluation Questions To what extent are the School-wide Activities in Figure 1 implemented in funded districts? To what extent do Local Circumstances in Figure 1 inhibit or facilitate implementation of School-wide Activities ? To what extent are School-wide Activities related to the Change Mechanisms in Figure 1? To what extent are School-wide Activities related to Outcomes in Figure 1? To what extent are Change Mechanisms related to Outcomes in Figure 1? 8

9 Evaluation Questions To what extent are the School-wide Activities in Figure 1 implemented in funded districts? What Local Circumstances do districts and schools find challenging? What are the levels of student outcomes in the funded and matched districts? 9

10 Evaluation Design 10

11 Early and Preliminary Findings Question 1: To what extent are the School-wide Activities in Figure 1 implemented in funded districts? Level of Implementation 11

12 Early and Preliminary Findings Question 1: To what extent are the School-wide Activities in Figure 1 implemented in funded districts? Assessment Knowledge 12

13 Early and Preliminary Findings Question 1: To what extent are the School-wide Activities in Figure 1 implemented in funded districts? Frequency of Assessment 13

14 Early and Preliminary Findings Question 1: To what extent are the School-wide Activities in Figure 1 implemented in funded districts? Frequency of Decision-making 14

15 Early and Preliminary Findings Question 2: What Local Circumstances do districts and schools find challenging? – Teachers and others tend to endorse practices that are RTI-aligned, but mostly when they are not described in terms of RTI – Relatively limited understanding on what RTI is and how it works, and a number of site visit participants had questions about the degree to which their “model” aligns with research or best practice – Implementation theory 15

16 Early and Preliminary Findings What are the levels of student outcomes in the funded and matched districts? – Percentage of students passing the 2008 WASL in Reading and Math, respectively – Baseline for ongoing comparison of changes in student outcomes 16

17 State Assessment - Reading 17

18 State Assessment - Math 18

19 Ongoing and Next Steps Small-scale study in one demo site (Walla Walla) Analysis of Year 2 data ( ) Collection of Year 3 data ( ) Refocusing program model and evaluation questions 19

20 1.Are you working with an evaluator in developing and/or monitoring implementation of RTI in your state? Feedback Pod Yes/No 20

21 “Cheshire,” Alice began rather timidly, “Would you tell me please, which way I ought to go from here?” “That depends a good deal on where you want to get to,” said the Cat. “I don’t much care-” said Alice. “Then it doesn’t matter which way you go,” said the Cat. Alice’s Adventures in Wonderland by Lewis Carrol Michigan Dept Ed. ppt

22 295 School Districts Educational Service Districts ESDs OSPI (SEA) Local Control Legislated as separate entities Special Ed SPDG Demo Districts State Leadership Team All special ed folks 22

23 Feedback Pod A. directly by the SEA. B. by other entities with direct guidance/training from the SEA to those entities. C. by other entities without coordinated influence from the SEA. 2. In my state, regional technical assistance and training is provided … 23

24 295 districts divided among 9 regional ESDs Leslie = ESD liaison for special ed CSASpokane* *Seattle Improvement planning - by region, using data profiles Root causes (linking activities to root causes) Quarterly updates Evaluation of activities Olympia* * May choose to focus some efforts on RTI and/or PBIS… 24

25 Federal State Local Education Agencies TA System Teachers/Staff Effective Practices System Alignment Fixsen,

26 Federal State (OSPI) LEAs TA System (ESDs) Teachers/Staff Effective Practices WA System Alignment Adapted from Fixsen, 2008 IHEs Teacher Prep/ Tech Asst Special Ed State Needs Projects 26

27 Other TA efforts --- State Needs Projects Needed a “system” to manage these projects and needed a system to provide TA to them.  to identify a “state need”  to ensure consistency/alignment  to provide training/support to the project staff Projects are supposed to address a “state need” -- Realized that there wasn’t a process -- 27

28 Contracted with UW-Tacoma to evaluate the State Needs Projects  Are they serving statewide?  Are they using resources/materials aligned with State policies and current research?  Are they providing on-going support?  Are they collecting reliable data?  Are they making adjustments based on evaluation of their efforts?  Are they having an impact on practice?  Are they having an impact on indicator performance? 28

29 Engage SEA departments/ESDs in strategic planning Multiple depts + ESDs = Implementation Team  NCRTI – NWRCC – WRRC  Need to develop common language  Need to align messaging/PD/use of coaches…  Need to develop review/vetting process for RTI  Develop strategic plan for support Build a sustainable State structure for RTI Build capacity of SEA, ESDs to support districts 29

30 295 School Districts Coordinated Services Agreement (CSA) w/ ESDs (regional TA providers) SEA/ESD Implementation Team Special Ed SPDG 30

31 OSPI Implementation Team - really should be called the Transformation Team Teaching & Learning (Reading, Math) Classroom Assessment District & School Improvement Special Education Migrant/bilingual/ELL Title I/LAP Highly Capable Equity & Civil Rights Secondary Ed & Dropout Prevention Student Achievement (Achievement Gap) Ed Technology Financial Resources & Governmental Relations (Data Governance) Educational Service Districts (ESDs) 31

32 This would begin to align many efforts at the SEA – which would also impact the ESDs… Dropout preventionDistrict & School Improvement & Accountability K12 Reading Model New Math framework Comprehensive Assessment System SPDG 32

33 Implementation Team developed an assessment tool: (OSPI Efforts Inventory) Assessing activities across SEA (pilot…11/12/10…two departments) In order to develop a strategic plan for the state, we needed to understand WHAT everyone was doing, and HOW IT CONNECTED to the RTI structure. 33

34 Focus Area Essential Components (RTI Framework) OSPI Efforts Multi-Level System Screening Progress Monitoring Data- Based Decision MakingDocuments Profes- sional Develop- ment Fund- ing Source Data Collected Core Instruction/ Support Interven- tions Reading Writing Mathematics Language Behavior Alcohol, Tobacco, or Other Drug Use Mental/ Emotional Health OSPI Efforts Inventory Department Contact Person(s) Date 34

35 Improved instruction for students 35

36 3. We have developed common language across the SEA and our resources are aligned and collectively support the RTI framework. Feedback Pod True/False 36

37  engaged IHEs in IRIS Center training  recruited IHE reps on State LT (gen & sped)  engaged multiple IHEs in regional training  meeting with several IHEs re: RTI efforts Build capacity of IHEs Build a sustainable State structure for RTI UW-T 325T grant – participating on Advisory Board Project RTI: Restructuring, Transforming, Implementing a Dual-track RTI Teacher Preparation Program 37

38 4. We have developed common language across the state (IHEs, SEA, regional TA providers) and our resources are aligned and collectively support the RTI framework. Feedback Pod True/False 38

39 Shewhart (1924); Deming & Juran (1948); Six-Sigma (1990) Plan – Decide what to do Do – Do it (be sure) Study – Look at the results Act – Make adjustments Cycle – Do over and over again until the intended benefits are realized PDSA Cycles: Trial & Learning Dean L. Fixsen & Karen A. Blasé,

40 “Knew”: Lack of understanding of pilot site requirements Lack of consistent PD opportunities Lack of a state structure for PD Lack of involvement of SEA beyond sped o lack of understanding about connections o lack of alignment o no vetting of materials Lack of common understanding of RTI framework Lack of data system to manage RTI efforts Lack of preparation for leadership of RTI efforts Likely a lack of fidelity 40

41 Evaluation results –  1-2 PD opportunities (avg)  More than half gave incorrect response or no response regarding defining screening & progress monitoring  Collecting data but teachers struggling to use data  Basic knowledge of RTI lacking  Lack of buy-in  Difficulty moving students through tiers  Difficulty in using staff to provide interventions  Is anyone monitoring fidelity? 41

42 So, need to develop a system to support demo sites and all others who want to implement RTI: Develop a state structure for support of RTI  develop common language  develop a strategic (state-level) plan  align efforts (messaging, resources, training,…) Purchase/develop a data system for demo sites Provide ‘hands on’ support to demo sites 42

43  Screening /progress monitoring  Selecting EBPs  Adapting on-site technical assistance  Developing monthly TA calls (in addition to quarterly mtgs)  Developing a Guide for Selection of EBPs  “ “ Guide for Selection of Trainers/materials  Regional training – IHEs, ESD, multiple SEA staff  Using integrity rubric Training for RTI demo sites 43

44 Build a data system WA received $17.3 million for longitudinal data system -- requested meeting with data folks (SEA) -- have been meeting with Student Information Director and Data Governance Coordinator (SEA) -- also invited DSIA, Classroom Assessment Spectrum K12 Demo for Implementation team Reviewing training plans Meeting to draw up a “proof of concept” system 44

45 Supports for Implementation RTI Coordinator mtgs (quarterly) Site visits (technical assistance) Will begin monthly contacts to provide additional supports Developing a series of trainings for all (will consist of face-to-face, webinar, and recorded sessions) 45

46 How did you use your data this past year? Student-level data Building-level data District-level data Protocols in place? How is your data system driving implementation in your district??? Questions for RTI Coordinators 46

47 5. My SEA used data this past year to: Feedback Pod A) improve content, delivery, format, or target audience of PD B) revise guidance on RTI framework C) influence policy-makers, TA providers, SEA divisions D) improve data collection E)determine funding efforts F) revise evaluation efforts G)obtain technical assistance/support H) other: _____________________ Check all that apply 47

48 Intensive Technical Assistance w/ the National Center on RTI (MOU) Common language Alignment across efforts Build capacity at the SEA, IHEs, ESDs, & districts to support implementation of evidence-based practices (EBPs) Data system for demonstration districts (expandable) Information dissemination system to expand RTI efforts across universities, districts and professional organizations 48

49 Effective State Support Structures for RTI Include… Leadership, consensus, and aligned vision Consistent messaging across departments Capacity-building supports for systems that support LEAs/schools (IHEs, ESDs), and for LEAs and schools directly Collection and use of data to inform decisions at all levels (student, classroom, school, LEA, SEA) Regulatory language, in general and/or special education, as appropriate, to support RTI From Innovations conference,

50 Invited to participate in the SISEP Community of Practice Putting together a team of folks from the SEA & ESDs from Implementation team - from CSA group Our CSA group is now looking at “scaling up evidence-based practices” 50

51 Presenting to Cabinet 11/22/10 Putting the puzzle together for them:  info on efforts of multiple depts ( including pilot data - OSPI Efforts Inventory )  connecting RTI to the Superintendent’s Five Strategic Priorities  # districts implementing across the state  CSA work >50% report implementing RTI 51

52 152 (51.5%) districts report implementing RTI 142 Reading – Elem (48.1%) 79 Reading – Middle (26.8%) 42 Reading – High School (14.2%) 98 Math – Elem (33.2%) 67 Math – Middle (22.7%) 37 Math – High School (12.5%) 69 Behavior – Elem (23.4%) 44 Behavior – Middle (14.9%) 26 Behavior – High School (8.8%) school year 52

53 At first, it felt like… 53

54 Each step we take has an impact on yet another area ---- Before long (that’s judged in “State” time ), we’ll have all of our state-level activities aligned… 54


Download ppt "Washington State Using Data to Drive Statewide Improvement Efforts Greg Roberts, Evaluation Research Services (ERS) Leslie Pyper, SPDG Director, Learning."

Similar presentations


Ads by Google