1. 2011 OSEP Leadership Mega Conference Collaboration to Achieve Success from Cradle to Career 2.0 State Monitoring Under IDEA A Snapshot of Past Practices and a Framework for the Future Julie Bollmer and Marsha Brauen, Westat Presentation EB 1

2. Study objectives and historical context Overview of framework for state monitoring systems Methodology for site visit data collection Discussion of each framework component Summary Questions/discussion 2 Overview of Presentation

3. Study O bjectives and Historical Context 3

4. Provide a description of the nature and scope of states’ monitoring systems Describe states’ monitoring systems at two points in time Create a framework to describe state monitoring systems 4 Study Objectives

5. State monitoring systems in 2004-05 and 2006-07 were the focus of the study. Systems were influenced by: – Federal legislation – OSEP’s monitoring systems – SPP and APR implementation. 5 Historical Context of the Study

6. F ramework for State Monitoring Systems 6

7. Development of Framework7 No requirement that states use a particular monitoring approach Framework used to help describe the variation in state monitoring systems Framework incorporated important activities of state monitoring systems Used various sources to create the framework Multiple reviews by Advisory Panel

8. 8 Framework for State Monitoring

9. Methodology for Site Visit Data Collection 9

10. Random sample of 20 states  Two rounds of data collection First round  2004-05 Second round  2006-07  Conducted semi-structured interviews with state staff, local staff, stakeholders  Reviewed states’ documentation on monitoring activities 10 Site Visit Data Collection

11. Single or Multiple Processes Number of Monitoring Processes and States with Multiple Monitoring Processes Number Part BPart C 2004-05 (N=20) 2006-07 (N=20) 2004-05 (N=20) 2006-07 (N=20) Monitoring processes across states34322428 States with more than one process10 37 11

12. Discussion of Each Framework Component 12

13. Problem Identification Problem Identification: Comparing performance (e.g., on a specific indicator) to an expectation (e.g., a target established for that indicator) and detecting possible deficiencies. – Indicator and Target Setting – Indicator Data Collection and Analysis – Problem Detection 13

14. Problem Identification14

15. Examples for Problem Identification: – Stakeholder committee includes parents of children with disabilities or representatives from advocacy groups. – Target setting is accomplished through a systematic process. – Written documentation exists describing the data collection methodologies, including site visit. – Procedures are in place to monitor the quality of the data collected. – Findings reflect performance in relation to specific targets. – State-level problem identification reports are disseminated to stakeholders. 15

16. Problem Identification Percentage of Part B and Part C Monitoring Processes That Had Problem Identification 16

17. 17 How indicators selected Part BPart C 2004-05 (N=34) 2006-07 (N=32) 2004-05 (N=24) 2006-07 (N=28) By state, with stakeholder input21854 By state, without stakeholder input138195 By state, based on SPP/APR†16†19 Problem Identification Number of Monitoring Processes That Used Various Approaches to Select Indicators † Not applicable. Because the 2004 amendments to IDEA, which require states to submit SPPs/APRs, were not enacted until December 2004, this approach was not used by states when selecting their indicators for 2004-05.

18. Problem Investigation Problem Investigation: Exploring why an identified problem exists and whether it is systemic or localized/isolated 18

19. Problem Investigation Examples for Problem Investigation: – Materials used to seek input from stakeholders are tailored to improve understanding and facilitate their contribution to the problem investigation process. – Individuals conducting problem investigations have appropriate training. – Procedures are in place to minimize interruptions/ disruptions to school, program, or district routines when collecting data. – Data are verified for accuracy. – Findings focus directly on the identified problems under investigation. 19

20. Problem Investigation Percentage of Part B and Part C Monitoring Processes That Had Problem Investigation 20

21. 21 Approach Part BPart C 2004-05 (N=16) 2006-07 (N=17) 2004-05 (N=8) 2006-07 (N=8) Same approach for all problems10 75 Tailored, based on nature of problem6713 Problem Investigation Number of Monitoring Processes in Which Various Problem Investigation Approaches Were Used

22. Corrective Action and Enforcement Corrective Action and Enforcement: Correcting noncompliance by making immediate changes to documentation, procedures, or practices 22

23. Corrective Action and Enforcement Examples for Corrective Action and Enforcement: – CAPs specify all problem areas. – Steps/components of the CAPs are clearly delineated for each identified problem. – Timelines are included for each step/component of the CAPs. – CAPs are disseminated to local stakeholders beyond the LEA/EIS program administrators. – Opportunities are provided for LEA/EIS program personnel to discuss CAPs with SEA/lead agency staff. – The SEA/lead agency has written general enforcement procedures. 23

24. Corrective Action and Enforcement Percentage of Part B and Part C Monitoring Processes That Had Corrective Action and Enforcement 24

25. 25 Follow-up approach Part BPart C 2004-05 (N=32) 2006-07 (N=32) 2004-05 (N=21) 2006-07 (N=23) General communication10941 Submit evidence of implementation51119 Progress reports211106 Site visits, meetings, data reviews9171114 None (did not follow up)8431 Corrective Action and Enforcement Number of Monitoring Processes in Which Various Follow-up Approaches Were Used to Ensure Corrective Action Plan Implementation

26. Improvement Planning and Implementation Improvement Planning and Implementation: Developing and implementing strategies for improving systemic performance or reducing the potential for noncompliance – Improvement Planning – Improvement Plan Implementation 26

27. Improvement Planning and Implementation27

28. Examples for Improvement Planning and Implementation: – Input is sought from parents of children with disabilities or representatives from advocacy groups. – Written improvement plans address each problem area. – Plans describe the ways that implementation will be monitored by the SEA/lead agency. – Documentation confirms that plans were followed or changes in plans justified. – Meetings are held with SEA/lead agency to discuss local implementation. 28

29. Improvement Planning and Implementation Percentage of Part B and Part C Monitoring Processes That Had Improvement Planning and Implementation 29

30. 30 Follow-up approach Part BPart C 2004-05 (N=23) 2006-07 (N=21) 2004-05 (N=11) 2006-07 (N=12) General communication6233 Submit evidence of implementation1302 Progress reports51467 Site visits, meetings, data reviews81066 None (did not follow up)8332 Improvement Planning and Implementation Number of Monitoring Processes in Which Various Follow-up Approaches Were Used to Ensure Improvement Plan Implementation

31. Reassessment Reassessment: Checking to see whether a corrective action plan or an improvement plan has been effective 31

32. Reassessment Examples for Reassessment: – Reports describe the data used to reassess performance/compliance on specified indicators and targets. – Opportunities are provided for LEA/EIS program personnel to discuss reassessment reports with SEA/lead agency staff. – Findings from reassessments are disseminated to state-level stakeholders. 32

33. Reassessment Percentage of Part B and Part C Monitoring Processes That Had Reassessment 33

34. 34 Reassessment Number of Monitoring Processes in Which Reassessments Were Conducted by LEA/EIS Program Staff and/or State Agency Staff Who conducted reassessments Part BPart C 2004-05 (N=11) 2006-07 (N=8) 2004-05 (N=12) 2006-07 (N=9) LEA/EIS program staff only3231 State agency staff only6657 Both LEA/EIS program staff and state agency staff 2041

35. Summary 35

36. Framework has not been evaluated to look at whether its adoption will improve compliance with IDEA and outcomes for students with disabilities; limits the conclusions that could be drawn from the study (e.g., did not look at change over time or trends). The framework is not the only one that could be developed to represent state monitoring systems. Retrospective information was collected 36 Limitations of the Study

37. State monitoring systems were not static; in both years, a percentage of them were in transition Many factors influenced states’ monitoring systems Substantial variability in the nature and design of states’ monitoring systems In both years, few Part B or C monitoring processes included all five framework components 37 Key Findings

38. In both years, Problem Identification and Corrective Action and Enforcement were the most common framework components included in state monitoring processes States varied in the way they carried out each component and the degree to which elements were present or absent 38 Key Findings

39. Reports from this study are available on the IES website: http://ies.ed.gov/ncser/pubs/ The database is restricted-use and will need to be obtained from IES Email: Marsha Brauen (marshabrauen@westat.com) Julie Bollmer (juliebollmer@westat.com) Rob Ochsendorf (robert.ochsendorf@ed.gov) 39 Want More Information?