1. SSIP Phase I: Data Analysis Part C/619 State Accountability Priority Area April 8, 2014

2. Disclaimer This SSIP presentation and supplemental materials were developed prior to OSEP’s publication of the final SPP/APR package

3. Webinar Goals Participants will leave the webinar with a basic understanding of: – Phase I: Data Analysis process – Resources and strategies that can support states in the Data Analysis process – Considerations for engaging stakeholders in the Data Analysis process 3

4. Data Analysis Requirements A description of how the state analyzed key data to: (1)select the State-identified Measurable Result. (2)identify root causes contributing to low performance. The description must include information about: (1)how the data were disaggregated by multiple variables. (2)if applicable, any concerns about the quality of the data and how the state will address these concerns. (3)If applicable, methods and timelines related to any additional data to be collected and analyzed. 4

5. Measurable Result Requirements May, but need not, be an SPP/APR indicator or a component of an SPP/APR indicator. Must be clearly based on the data and state infrastructure analyses. Must be a child-level outcome in contrast to a process outcome. May be a single result or a cluster of related results. 5

6. Baseline and Targets Requirements States must establish baseline (expressed as a percentage) with the Measurable Result (FFY 2013 data) States must set targets (expressed as percentages) for each of five years FFY 2014 – FFY 2018 FFY 2018 target must be higher than FFY 2013 baseline 6

7. Fundamentals of Data Analysis EIA – Evidence – Inference – Action Starting with questions 7

8. Fundamentals of Data Analysis EIA – Evidence: just the #s – Inference: interpretation – Action: steps to be taken 8

9. Starting with Question(s) Where are areas of lower performance? (analysis by variables) Geographic areas of the state Child/family characteristics Program characteristics 9

10. Starting with Question(s) Where are areas of lower performance? (analysis by child characteristics) – Does our program serve some children more effectively than others? » Do outcomes vary for children with different racial/ethnic backgrounds? » Are outcomes different for Dual Language Learners as compared to mono-language learners? 10

11. Starting Points Potentially starting with: An Issue (e.g. shifting demographic) An Initiative Child outcomes data 11

12. Questions and Existing Initiatives What is the state performance in social emotional development compared to other outcome areas? What have our trends been in the area of social emotional development for young children? Are there certain areas of the state that have lower performance in the area of social emotional development for young children? Do social emotional outcomes differ by child characteristics (race/ethnicity, socioeconomic status, geographic region of the state)? 12

13. Broad Data and Infrastructure Analysis Purpose – Explore child results (and potentially the related family results) and practices that would be justifiable targets for improvement Goal – Assemble evidence to substantiate to leadership and stakeholders why you selected a particular result Strategies – Analysis of child results and related data to identify areas of lower performance 13

14. Types of Broad Data Analysis Analysis of child outcomes data By summary statement State data compared to national data Local data comparisons across the state State trend data Analysis by race/ethnicity, disability, income Analysis of related family outcomes data State data compared to national data Local data comparisons across the state State trend data Analysis by race/ethnicity, income, length of time in program Linked to child outcomes data

15. Resource: Broad Data Analysis Template Purpose: to look at how children in the state are performing relative to national data, across years, within the state and by comparisons across programs 15 http://ectacenter.org/eco/assets/docs/SSIP_child_outcomes_broad _data_analysis_template_FINAL.docx

16. Resource: Broad Data Analysis Template 16

17. Resource: Broad Data Analysis Template 17

18. Meaningful differences calculator Purpose: to look at statistical significance of change in state SS data from year to year; and allow comparison of local to state 18 http://ectacenter.org/eco/pages/summary.asp#meaningfuldiffcalc

19. Resource: Analyzing Child Outcomes Data for Program Improvement Quick reference tool Consider key issues, questions, and approaches for analyzing and interpreting child outcomes data. 19 http://www.ectacenter.org/~pdfs/eco/AnalyzingChildOutc omesData-GuidanceTable.pdf

20. Guidance Table 20

21. What is the likely child result that will be the focus of your SSIP? Social Relationships Knowledge and Skills Action to Meet needs All three of the above Something other than above 21

22. In-Depth Data and Infrastructure Analysis Purpose – Conduct further analysis exploring the link between the practices and infrastructure and the child result. Goal – Gather sufficient evidence to link specific practices and infrastructure to child results (to inform needed improvement strategies). Strategies - Subgroup analysis, comparisons of programs, “root cause analysis,” local data drill- down, narrative summary of analysis 22

23. Identify Root Causes Contributing to Low Performance Analyze data at the local level Identify factors contributing to low performance (including infrastructure) Contributing factors: – Explain why you have the problem – Point to how the problem can be addressed 23

24. Identify Root Causes Contributing to Low Performance Identify barriers for each contributing factor – What is standing in the way of addressing this contributing factor? – Why hasn’t it been addressed to date? 24

25. Resource: Subgroup Analysis Template Purpose: to provide states with table shells for subgroup analyses that have proven useful in understanding predictors of child outcomes. 25 http://ectacenter.org/eco/pages/usingdata.asp

26. Subgroup Analysis Example 26

27. Local Contributing Factor Tool 27 http://ectacenter.org/~docs/eco/ECO-C3-B7-LCFT.docx http://ectacenter.org/~docs/topics/gensup/14-ContributingFactor- Results_Final_28Mar12.doc

28. LCFT: Question Categories System/ Infrastructure Practitioner/ Practices Policies/ procedures Funding Training/TA Supervision Data Personnel Competencies of staff Implementation of effective practices Time Resources Supports 28

29. Data Quality Not the focus of the SSIP But must be addressed in the SSIP – Describe data quality issues – Describe data quality improvement efforts

30. Data Quality: Pattern Checking Checking predictable patterns to help determine ‘red flags’ to be investigated in the data. http://ectacenter.org/eco/assets/pdfs/Pattern_Checking_Table.pdf

31. Data Quality Profiles The profiles include information about: – State vs. national – Data quality criteria used for national analysis Completeness of data Progress categories patterns – Trends over time Contact: Abby Winer abby.winer@sri.comabby.winer@sri.com

32. Data Analysis State Example Virginia Part C Beth Tolley Kyla Patterson

33. Starting Point Child and family outcome data Stakeholder Input – State Interagency Coordinating Council (VICC) – Local System Managers

34. Preparation Child Outcomes Broad Data Analysis Template Data Quality Profile Support from ECTA and DaSy

35. Process Overview of SSIP Powerpoint presentations and handouts with data and analysis questions Large and small group discussion and input

36. Broad Analysis Questions Does our state’s data look different than the national data? Are our state outcomes trends stable over time? – Is the data trending upwards? – Is the data trending downwards? Is our state performing more poorly in some outcomes than others? Are the outcomes similar across programs? What about data quality? Can we be confident in our data?

37. Child Outcomes: National vs. State FFY11 and State FFY12

38. National Vs. State Meaningful Differences

39. Virginia Trends

43. Child Outcomes: Local vs. State

44. Data Quality Elements – Completeness of data number of children reported for the outcome/number who exited Virginia’s results: average= 65%; range for Local Systems = 17% - 100% – Expected Patterns for Progress Categories Virginia’s state date is within these parameters for all three outcomes – Child Outcomes State Trends Over Time As noted on previous slides, Virginia’s results do not show wide variations which would trigger concerns about data quality

46. Family Outcomes: State Trends over Time

47. Family Outcomes: Local vs. State 2012-2013 4C: EI has helped the family help their child develop and learn

48. Identification of Area of Concern Based on broad data analysis of child and family outcome data VICC and LSM identified same area of concern: – Outcome 3C – Use of appropriate behaviors (taking action) to meet needs

49. In-Depth Data Analysis Purpose: To identify the specific measurable result To identify root causes and contributing factors – why is it happening?

50. Stakeholder Questions Does the child’s reason for eligibility impact results on this outcome? Does age at entry or length of time in EI impact results? Does use of Part B entry ratings as Part C exit ratings have an impact? Is there consistent understanding of the developmental areas involved in determining a rating on this indicator?

51. Next Steps Disaggregate data by child characteristics – Age at entry – Length of time in service – Race/ethnicity – Reason for eligibility Analyze data by local system Continue to ask why questions

52. Methods National Resources, such as: – SSIP subgroup analysis template – Analyzing child outcomes for program Improvement Joint analysis with Local System Managers through regional meetings Back to the VICC in June

53. Section 619 and the SSIP Some considering: 619 incorporated into the Part B 619 focus for the Part B Part C and Section 619 Coordination across 0-21 53

54. Richard Henderson Evelyn S. Johnson Shannon Dunstan Early Childhood and Interagency Coordinator Division of Special Education Division of Student Achievement and School Improvement Idaho State Department of Education

55. Idaho State Systems Improvement Plan Focus Shifting to Performance K-12 Priorities Indicator 3: Participation and Performance on Statewide Assessments Indicator 5: Participation/Time in General Education Settings (LRE) Indicator 14: Post School Outcomes

56. Big B Priorities Indicator 3 Indicator 5 619 Alignment Indicator 7 Indicator 6

58. Social and Emotional Development Parents School Districts Head Start Programs Child Care Providers

59. The Protective Factors Framework Parental Resilience Social Connections Concrete Support in Times of Needs Knowledge of Parenting and Child Development Social and Emotional Competence of Children

60. Statewide Systematic Implementation of Social and Emotional training across Programs

61. Other activities Send out a Statewide surveys to: Special Education Preschools, Head Start, and Child Care Providers to define need and readiness Early Childhood Coordinating Council has adopted this as priority for Head Start subcommittee Attend a day long statewide planning meeting as preconference to NAEYC

62. Contact Information: Shannon Dunstan Early Childhood & Interagency Coordinator Idaho State Department of Education Division of Student Achievement and School Improvement Division of Special Education (208) 332-6908 sdunstan@sde.idaho.gov

63. Summary Fundamentals of Data Analysis Starting with a question (or questions) Evidence, Inference, Action process Broad data Analysis To explore child results and related family results and practices that would be justifiable targets for improvement Resources: Broad Data Analysis Template, Analyzing Child Outcome Data, Meaningful Differences Calculator In-depth Data Analysis To look at local level data and identify causes of low performance Resources: LCFTs, Subgroup Analysis Data Quality Describe concerns and how the state will address these concerns. Resources: State Profiles, Pattern Checking Document

64. Contact Information Christina Kasprzak, ECTA/DaSy Christina.Kasprzak@unc.edu Cornelia Taylor, ECTA/DaSy cornelia.taylor@sri.com Abby Winer, ECTA/DaSy Abby.winer@sri.com Anne Lucas, WRRC/ECTA Anne.Lucas@unc.edu Megan Vinh, WRRC Mvinh@uregon.edu 64

65. Thank you for your attention! This is the second in a series on SSIP presented in 2014. Resources related to this call and other presentations in the series are available at the following URL: http://ectacenter.org/~calls/2014/ssip/ssip.asp IDEA DATA CENTER