1. Data-based practitioners: How to use data for decision making Megan Vinh, PhD Abby Winer Schachner, PhD 13 th National Training Institute on Effective Practices: Addressing Challenging Behavior April 22, 2016

2. 2 Welcome Who’s in the room?  Who are you affiliated with?  What areas do you work in?

3. Agenda National context Key concepts for data-based decision-making Hands-on activity Share-out & Reflections Wrap-up

4. National Context: Results Driven Accountability For over 30 years, there has been a strong focus on regulatory compliance based on the IDEA and Federal regulations for early intervention and special education  OSEP  States  Districts/Programs As a result, compliance has improved!

5. National Context: Why Now? Despite this focus on compliance, states are not seeing improved results for children and youth with disabilities:  Young children are not coming to Kindergarten prepared to learn  In many locations, a significant achievement gap exists between students with disabilities and their general education peers  Students are dropping out of school  Many students who do graduate with a regular education diploma are not college and career ready Michael Yudin

6. National Context: Results Driven Accountability OSEP has revised its accountability system to shift the balance from a system focused primarily on compliance to one that puts more emphasis on results. A key component is the State Systemic Improvement Plan

7. National Context Three joint policy statements have been released:  Policy Statement on Inclusion of Children with Disabilities in Early Childhood Programs  Policy Statement on Expulsion and Suspension Policies in Early Childhood Settings  Policy Statement on Family Engagement

8. What do these all have in common? They all require the use of data to inform planning and to systemically improve results for children with disabilities and their families.

9. Ways to Use Data Identifying issuesMonitoring System or Local planning Determining Improvement activities System oversight/management Adapted from Building local capacity for data analysis Data Accountability Center (DAC) webinar

10. Key Concepts for Data-Based Decision-Making What are your questions? What is your process for looking at data and making interpretations? What are the data sources you might have? Is there other data you need to collect or gather?

11. Starting with a question (or two…) All analyses are driven by questions Questions come from different sources Different versions of the same question are necessary and appropriate for different audiences. What are your critical questions?

12. Defining Data Analysis Questions What are your crucial policy and programmatic questions? Example: 1. Does our program remove some children more often than others? a. Are children with different racial/ethnic backgrounds removed at similar rates?

13. Sample Basic Questions How many children are being removed (suspended or expelled)? Who is being removed? How many children are receiving behavior incidence reports? Who is accessing behavior or mental health consultation? How many children are receiving comprehensive evaluation services?

14. Sample questions that cut across components How do removals relate to child characteristics? Which children are being removed the most? Are some classrooms removing children more than others? What factors are related to higher removals?

15. 15 Practitioners/Teachers

16. 16 Local Early Intervention Service (EIS) Program / Local Education Agency (LEA)

17. What is Your Process for Looking at Data? Evidence Inference Action 17

18. 18 Evidence Evidence refers to the numbers, such as “31% of children have been removed at least once” The numbers are not debatable

19. 19 Inference How do you interpret the evidence? What can you conclude from the numbers? Does evidence mean good news? Bad news? News you can’t interpret? To reach an inference, sometimes you need to analyze data in other ways (ask for more evidence) 19

20. 20 Inference Inference is debatable -- even reasonable people can reach different conclusions Stakeholders and having a variety of perspectives can help with putting meaning on the numbers Early on, the inference may be more a question of the quality of the data

21. 21 Action Given the inference from the numbers, what should be done? Recommendations or action steps Action can be debatable – and often is Another role for stakeholders and teams May involve looking at additional data and information Again, early on the action might have to do with improving the quality of the data

22. Preparation – Plan to Succeed Define purpose and the issue Identify who needs to be involved Timelines Identify relevant questions Identify relevant data Generate hypotheses

23. WHAT ARE YOUR DATA SOURCES? Data: facts or information used usually to calculate, analyze, or plan something

24. Cultural Barriers to Data-Based Decision Making 1.Many providers/teachers have developed their own personal metric for judging the effectiveness of their intervention/teaching and often this metric differs from the metrics of external parties (e.g., state accountability systems and school boards). 2.Many providers/teachers and administrators base their decisions on experience, intuition, and anecdotal information (professional judgment) rather than on information that is collected systematically. 3.There is little agreement among stakeholders about which child outcomes are most important and what kinds of data are meaningful. 4.Some providers/teachers disassociate their own performance and that of children, which leads them to overlook useful data. Ingram, D. S. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. Teachers College Record, 106(6), 1258–1287.

25. Technical Barriers Data-Based Decision Making 5.Data that providers /teachers want – about “really important outcomes” – are rarely available and are usually hard to measure. 6.Programs and schools rarely provide the time needed to collect and analyze data. Ingram, D. S. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. Teachers College Record, 106(6), 1258–1287.

26. WHAT ARE YOUR BARRIERS TO CREATING A CULTURE OF DATA- BASED DECISION MAKING? AND WHAT ARE POTENTIAL SOLUTIONS?

27. Small Group Activity Dilemma Evidence: Critically examine the data provided Inference: Discuss what inferences you can make Action: Brainstorm potential actions and next steps

28. Wrap-Up What were your inferences and actions based on the data? Did you have any ah-has? Reflections on the process and experience?

29. 29 Important Points for Being Successful Think about how you can maximize data you already collect and collect what you need Think about how to organize your staff and your agency around ongoing data use Its all about continuous improvement Use data to determine priority for focus It is important to “drill down” to understand performance to identify meaningful solutions

30. Sharing Your Results Communicate your data in a way that is appropriate for your audience based on how you expect them to use it Who are you communicating with? What is the key information that they need to know? When do they need the information? What other types of information do they need to help them understand the data? Think about the different ways you want to visualize present the data

31. Keeping In Touch Abby Schachner, abby.schachner@sri.com abby.schachner@sri.com Megan Vinh, mvinh@email.unc.edu mvinh@email.unc.edu DaSy Center website: http://dasycenter.org/ http://dasycenter.org/ ECTA Center website: http://ectacenter.org/ http://ectacenter.org/

32. Resources Joint Policy Statements  http://www.acf.hhs.gov/programs/ecd/child-health-development/reducing-suspension-and- expulsion-practices http://www.acf.hhs.gov/programs/ecd/child-health-development/reducing-suspension-and- expulsion-practices  http://www2.ed.gov/policy/speced/guid/earlylearning/joint-statement-full-text.pdf http://www2.ed.gov/policy/speced/guid/earlylearning/joint-statement-full-text.pdf DaSy Critical Questions for Early Intervention and Early Childhood Special Education  http://dasycenter.org/critical-questions-about-early-intervention-and-early-childhood-special- education/ http://dasycenter.org/critical-questions-about-early-intervention-and-early-childhood-special- education/ Planning, Conducting, and Documenting Data Analysis for Program Improvement  http://dasycenter.org/planning-conducting-and-documenting-data-analysis-for-program- improvement/ http://dasycenter.org/planning-conducting-and-documenting-data-analysis-for-program- improvement/ Head Start Modules on Creating a Culture that Embraces Data  http://eclkc.ohs.acf.hhs.gov/hslc/tta-system/operations/data/guide/guide.html http://eclkc.ohs.acf.hhs.gov/hslc/tta-system/operations/data/guide/guide.html

33. The contents of this presentation were developed under a grant from the U.S. Department of Education, # H373Z120002, and a cooperative agreement, #H326P120002, from the Office of Special Education Programs, U.S. Department of Education. However, those contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government. DaSy Center Project Officers, Meredith Miceli and Richelle Davis and ECTA Center Project Officer, Julia Martin Eile.