1. Identifying Meaningful Differences between State and Local Summary Statement Values Cornelia Taylor (DaSy, ECO); Mary Beth LaSalle (NH TA Consultant); Ruth Littlefield (NH 619 Coordinator); Iandia Morgan (NV Part C); Marcia Boswell Carney (KS Program Analyst) September 2013

2. Motivation for developing the meaningful differences calculator Theoretical basis Demonstration of the calculator Panel discussion on the uses

3. Motivation for development Identify meaningful differences –Year to year in a state –Local from state Evaluate the precision of numbers particularly in local areas with small numbers of children

4. Two functions of the calculator 1.Computes the difference between two summary statement values. 2.Calculates a confidence interval around a summary statement. 4

5. Examining confidence intervals around summary statements There is always some degree of fluctuation in measurement When we talk to stakeholders we want to know how to take that fluctuation into account. –If we expect a lot of fluctuation we want to communicate that so that stakeholders are not surprised if they see big changes from year to year. 5

6. THEORETICAL APPROACH 6

7. Precision of numbers The lack of precision in a number comes from measurement error and sampling error. –Measurement error differs across local areas and is estimated through studies of the reliability and validity that describe the accuracy of measurement. –Sampling error occurs even in perfect measurement and changes based on the number of children included in the estimate. In this discussion we are talking only about sampling error

8. Sampling error When we compute a summary statement value for a program, this value is one of many that could have resulted given the same kids and same program and same instrument. The degree to which an outcome measurement might change when the same kids participated in the same programs and were measured using the same process is important to know

9. Computing a probable range Information about sampling error can be used to compute a range around a summary statement that describes the range of probable values –Confidence interval – this range is often called a confidence interval because we can be confident that the true value lies within that range. 9

10. Amount of error by N size (2 – 100) 10

11. Amount of error by N size (100 – 600) 11

12. Amount of error in a percentage computed on 45 children

13. DEMONSTRATION 13

14. Panel Discussion What was the need in your state that was addressed through the use of confidence intervals?

15. How did you use the information from the confidence interval calculator? 15

16. What were the positive results of using the confidence interval? 16

17. What were the negative results of using confidence intervals? 17

18. How do you plan to use confidence intervals in the future? 18

19. Thank you! 19 You can find more information at our website http://projects.fpg.unc.edu/~eco/index.cfm