1. Measuring EC Outcomes DEC Conference Presentation 2010
Cornelia Taylor, ECO
Christina Kasprzak, ECO/NECTAC
Lisa Backer, MN DOE

2. Background

3. Government Performance and Results Act (GPRA)
Program Assessment Rating Tool (PART)
Individuals with Disabilities Education Act (IDEA)
Three federal driving forces

4. PART Review Findings for Part C and 619
Results not demonstrated:
“While the program has met its goal relating to the number of children served, it has not collected information on how well the program is doing to improve the educational and developmental outcomes of preschool children/infants and toddlers served.”
Read more at ExpectMore.gov
More about PART – with the website for expectmore.gov
[Notes from previous presentations – ‘Depending on your audience, there may need to be further explanation about what PART, Part C, and Section 619 are (and how they are known in the state) and perhaps expansion of discussion and issues for a more general early childhood audience if information on outcomes is being collected by a broader group within the state.’]

5. More on PART
Previous presentation -- PART programs examined in 2002; 50% programs had no performance data
Programs were looking at inputs, not results
Part C and Section 619
No long-term child outcome goals or data
Need to develop a strategy to collect annual performance data in a timely manner

6. How Office of Special Education (OSEP) Responded to PART
Required states to submit outcome data in their State Performance Plan (SPP) and Annual Performance Report (APR)
Funded the Early Childhood Outcomes (ECO) Center in October 2003 to gather input, conduct research, make recommendations, and assist states

7. Goal of early intervention/early childhood special education
“…To enable young children to be active and successful participants during the early childhood years and in the future in a variety of settings – in their homes with their families, in child care, in preschool or school programs, and in the community.”
(from Early Childhood Outcomes Center,
Under ECO, stakeholders got to work on the child outcomes, including this overarching goal for EI/ECSE

8. Part C Family Indicator
% of families who report that EI services have helped their family:
1. Know their rights;
2. Effectively communicate their children's needs; and
3. Help their children develop and learn.
2004-early 2005: ECO generates discussion and gathers input on child and family outcomes. Stakeholders came up with five family outcomes. These are the 3 OSEP decides to require in by EI programs.

9. Part C &619 Child Outcomes
1. Positive social-emotional skills (including social relationships); 2. Acquisition and use of knowledge and skills (including early language/communication [and early literacy]); and 3. Use of appropriate behaviors to meet their needs
2004-early 2005: ECO generates discussion and gathers input on child and family outcomes. Stakeholders came up with these three child outcomes. OSEP decides to require these for EI and preschool programs to report.

10. OSEP Reporting Categories
Percentage of children who:
a. Did not improve functioning
b. Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers
c. Improved functioning to a level nearer to same-aged peers but did not reach it
d. Improved functioning to reach a level comparable to same-aged peers
e. Maintained functioning at a level comparable to same-aged peers
3 outcomes x 5 “measures” = 15 numbers

11. Summary Statements
Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they turned 6 years of age or exited the program.
The percent of children who were functioning within age expectations in each Outcome by the time they turned 6 years of age or exited the program.
The APR reporting requires states to report how they are doing on specific indicators and also to set targets for improvement. OSEP decided (with ECO and state input) that setting targets on the 5 a-e categories across 3 outcomes (15 numbers total) was too complicated, especially given the interrelationship between the a-e categories (they add up to 100%, so if one goes up another goes down). So, the ECO Center worked with OSEP and states to determine what kinds of summary data states might set targets on. These are the 2 summary statements finally selected by OSEP for all programs to report on.

12. Summary Statements
Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program.
c + d___
a + b + c + d
For each of the 3 outcome areas, states must report on children who substantially increased their rate of growth. The numerator includes ‘c’ the children who narrowed the gap and ‘d’ the children who closed the gap. Notice the denominator does not include the ‘e’ category which are the children who came in functioning at age level in that outcome area.
Don’t forget, these outcome areas are measured and reported independently. So, a child may enter the program performing at age expectations in one or more of the areas.

13. Summary Statements
The percent of children who were functioning within age expectations in each Outcome by the time they exited the program.
d + e__
a + b + c + d + e
For each outcome area, a state must report on the percent of children who were functioning within age expectations by the time they exited the program. The numerator includes ‘d’ the children who closed the gap and ‘e’ the children who entered the program at age expectation.
Some have likened this summary statement to the ‘readiness or ready to learn’ concept.

14. Connecting the Child Outcomes Data
State Approach (COSF ratings /Assessment scores)
a-e progress categories
summary statements

15. Why Collect Outcomes Data? At the State and Local Levels
Purpose
To have data for program improvement
and
to respond to federal reporting requirements
To respond to federal reporting requirements
Value -- not just for federal reporting, but also for program improvement purposes

16. Need for Aggregated Data
At both state and local levels:
To document program effectiveness
To improve programs
Identify strengths and weaknesses
Allocate support resources, such as TA
What you can do with outcomes data that can be aggregated at the state and local levels.

17. System for Producing Good Child and Family Outcomes
Adequate funding
Good outcomes for children and families
Good Federal policies and programs
High quality services and supports for children 0-5 and their families
Good State policies and programs
Good Local policies and programs
Data are an essential part of the system for producing good child and family outcomes. The vision: Using data as a tool for program improvement. Delaware EI and ECSE programs will have quality data available on an ongoing basis about multiple components of the system
Outcomes for children and families
Programs and services provided
Personnel (types, qualifications, etc.)
Etc.
Strong Leadership
Prof’l Development
Preservice
Inservice

18. State Approaches

19. State Approaches to Measuring Family Outcomes
NCSEAM* Family Survey
26 states (46%)
ECO Family Outcomes Survey
21 states (38%)
State-developed survey
6 states (11%)
Combination of above
3 states (5%)
*National Center for Special Education Accountability Monitoring

20. State Approaches to Family Outcomes Measurement* – Part C Program
Early Childhood Outcomes Center – August 2010 MP GU
Legend:
ECO Family Outcomes Survey
State-developed survey
NCSEAM survey AS HI
*This map shows the approaches used to measure the three family outcomes for APR reporting on Indicator C4. Some states used additional tools/ approaches to measure other family variables.

22. State Approaches to Measuring Child Outcomes
Child Outcomes Summary Form (COSF)
41 (73%) Part C
Single assessment statewide
7 (13%) Part C
Publishers’ online assessment systems
3 (5%) Part C
Other approaches
5 (9%) Part C
According to the Annual Performance Reports submitted to OSEP in February 2010, most states were using the COSF. Some states have mandated that all local programs use the same tool for outcomes measurement. Some require everyone to use one assessment tool statewide e.g. the BDI, others the AEPS, etc. A few states are working with the publishers of assessment tools to develop electronic systems so that local providers enter data into the system and the computer generates a report (Creative Curriculum; AEPS; Highscope COR)

23. State Approaches to Child Outcomes Measurement – Part C Program
Early Childhood Outcomes Center –August 2010 MP GU
Legend:
COSF
Publishers’ on-line systems
One tool statewide
Other AS HI

24. Looking at the data across outcomes – Part C

25. State Approaches to Measuring Child Outcomes
Child Outcomes Summary Form (COSF)
36 (61%) 619
Single assessment statewide
9 (15%) 619
Publishers’ online assessment systems
6 (10%) 619
Other approaches
7 (12%) 619
*one state preschool program still unknown
According to the Annual Performance Reports submitted to OSEP in February 2010, most states were using the COSF. Some states have mandated that all local programs use the same tool for outcomes measurement. Some require everyone to use one assessment tool statewide e.g. the BDI, others the AEPS, etc. A few states are working with the publishers of assessment tools to develop electronic systems so that local providers enter data into the system and the computer generates a report (Creative Curriculum; AEPS; Highscope COR)

26. State Approaches to Child Outcomes Measurement – 619 Programs
Early Childhood Outcomes Center – August 2010 MP GU MH PW FM
Legend:
COSF
Publishers’ on-line systems
One tool statewide
Other AS HI

27. Looking at the data across outcomes – 619.

28. What states are doing now
Continuing training and TA on data collection system
Enhancing data systems
Developing data analysis
Identifying and addressing data quality issues
Identifying areas for program improvement

29. Keeping our eyes on the prize: High quality services for children and families that will lead to good outcomes.
The bottom line -- we collect and use outcomes data to improve services for children and families in our programs.