Early Childhood Outcomes: Using Data for Program Improvement Kathy Hebbeler ECO at SRI International Robin Rooney and Christina Kasprzak ECO at Frank Porter Graham Institute Illinois October, 2008

Objectives Understand the purposes of the child outcomes data collection Be familiar with key considerations related to accurately completing the Child Outcomes Summary Form Be able to evaluate child outcomes data for accuracy as part of supervision Understand the use of child outcomes data for program improvement, including for instruction, resource allocation, and professional development Early Childhood Outcomes Center

Why are we doing this? Early Childhood Outcomes Center

Keeping our eye on the prize: High quality services for children and families that will lead to good outcomes. Early Childhood Outcomes Center

Goal of early childhood special education “…To enable young children to be active and successful participants during the early childhood years and in the future in a variety of settings – in their homes with their families, in child care, in preschool or school programs, and in the community.” (from Early Childhood Outcomes Center, http://www.fpg.unc.edu/~eco/pdfs/eco_outcomes_4-13-05.pdf) Early Childhood Outcomes Center

High Quality Data on Outcomes Data are a piece of a system that helps to achieve overarching goals for children and families Data yield Evidence that allows you to make an inference that should lead to specific actions to improve the system. Early Childhood Outcomes Center

System for Producing Good Child and Family Outcomes Adequate funding Good outcomes for children and families Good Federal policies and programs High quality services and supports for children 0-5 and their families Good State policies and programs Good Local policies and programs Strong Leadership Prof’l Development Preservice Inservice

The Vision: Using Data as a Tool for Program Improvement Illinois will have quality data available on an ongoing basis about multiple components of the system Outcomes for children and families Programs and services provided Personnel (types, qualifications, etc.) Etc. Early Childhood Outcomes Center

Driving Force for Data on Child Goals Comes from the Federal Level Government Performance and Results Act (GPRA) Program Assessment Rating Tool (PART) Individuals with Disabilities Education Act (IDEA) Early Childhood Outcomes Center

Government Performance and Results Act (GPRA) passed in 1993 Requires goals and indicators be established for IDEA Indicators and data collection further along for school age population than for EC Previously, for early childhood data had been collected on: Number of children served Settings Notes from OSEP’s slides: Lou GPRA: Passed in 1993 Applies to all federal agencies Requires that a set of indicators be established in order to judge the effectiveness of programs meeting their goals OSEP has established GPRA indicators for all IDEA programs, including Parts B, C, and D that are reported to Congress annually. For the Part C indicators, OSEP has collected data on the total number of children served and the number of children served under the age of one, and on settings where services are provided to eligible children For 619, OSEP has collected data on settings where children receive special education and related services Thus far OSEP has not yet collected and reported data on the functional abilities of children and family outcomes Early Childhood Outcomes Center

PART evaluation results (2002) 130 programs examined in 2002; 50% programs had no performance data Programs looking at inputs, not results Part C and Section 619 No long-term child outcome goals or data Need to develop a strategy to collect annual performance data in a timely manner During the 2003 budget period, the Administration rated approximately 130 federal programs on their effectiveness using part 50% of the these programs had no performance data Programs tend to use inputs or process data, rather than results or outcome data Specifically for Part C and Section 619: did not do well in the area of accountability/results because there were no long-term child outcome goals; no child outcome data. The recommendation from the PART is that OSEP needs to develop a strategy to collect annual performance data (child and family outcomes) in a timely manner Early Childhood Outcomes Center

Federal Funding for Preschool Special Education Total U.S. Illinois 2004 $387,699,000 $17,943,956 2005 $381,385,691 $17,812,936 2006 $380,751,030 $17,650,453 Early Childhood Outcomes Center

Early Childhood Outcomes Center

Individuals with Disabilities Education Act SEC. 616. <<NOTE: 20 USC 1416.>> MONITORING, TECHNICAL ASSISTANCE, AND ENFORCEMENT. ``(a) Federal and State Monitoring.-….. ….. ``(2) Focused monitoring.--The primary focus of Federal and State monitoring activities described in paragraph (1) shall be on-- ``(A) improving educational results and functional outcomes for all children with disabilities; Early Childhood Outcomes Center

How Office of Special Education (OSEP) responded Required states to submit outcome data in their Annual Performance Report (APR) Funded the Early Childhood Outcomes (ECO) Center to do research, make recommendations, and assist states Early Childhood Outcomes Center

Federal reporting requirements Where are we now: Federal reporting requirements Early Childhood Outcomes Center

OSEP Reporting Requirements: the Outcomes Positive social emotional skills (including positive social relationships) Acquisition and use of knowledge and skills (including early language/ communication [and early literacy]) Use of appropriate behaviors to meet their needs Early Childhood Outcomes Center

OSEP Reporting Categories Percentage of children who: a. Did not improve functioning b. Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c. Improved functioning to a level nearer to same-aged peers but did not reach it d. Improved functioning to reach a level comparable to same-aged peers e. Maintained functioning at a level comparable to same-aged peers 3 outcomes x 5 “measures” = 15 numbers Early Childhood Outcomes Center

Reporting Schedule Due February 2008 Data in reporting categories at exit for all children who have been in the program for at least 6 months Must be reported for the year beginning July 1, 2006 Repeat with next year’s data in 2009, etc. 2010 States must set targets Summary statements for targets recommended; not yet official* States must report data to public by school districts for these target numbers *See http://www.fpg.unc.edu/~ECO/meetings.cfm for more information on recommended targets Early Childhood Outcomes Center

State decisions and activities Where are we now: State decisions and activities We are focusing the remaining discussion on states activities – because this work is unfolding in states. Early Childhood Outcomes Center

WHY? Purpose (State Version) To have data for program improvement and to respond to federal reporting requirements To respond to federal reporting requirements Early Childhood Outcomes Center

WHY? Purpose (Local Version) To have data for program improvement and to provide data to the state To provide data to the state Early Childhood Outcomes Center

State approaches Most states have embraced outcomes measurement and are collecting outcomes data for their own purposes. Many states are building bigger systems than needed to produce the federal data. Go to www.the-eco-center.org for more information about what other states are doing Early Childhood Outcomes Center

How are states collecting child outcomes data? Possible state approaches to collection of child data Child Outcomes Summary Form (COSF) Publisher’s online assessment system Single assessment statewide Other approaches Early Childhood Outcomes Center

State Approaches to Measuring Child Outcomes Part C (56 states) Preschool (59 states) One tool statewide 8/56 (14%) 13/59 (22%) Multiple Publishers’ online tools 2/56 (4%) 3/59 (5%) COSF 7 pt. scale 40/56 (71%) 36/59 (61%) Other 6/56 (11%) 7/59 (12%)

Checking in – How is it going? Implementing the Child Outcomes Summary Form Process What is going well? What is not going well? Are there other issues or concerns that need to be addressed about the process? Early Childhood Outcomes Center

Child Outcome Summary Form Let’s Review: Child Outcome Summary Form Early Childhood Outcomes Center

Learning from Each Other Did you attend a training on the COSF? Have you participated in a COSF team rating process? For more than 10 children? 20? 30? Have you reviewed COSF forms completed by others? Have you provided training on the COSF? How confident are you in your knowledge of the COSF process? Early Childhood Outcomes Center

Essential Knowledge for Completing the Child Outcomes Summary Form Between them, team members must: Know about the child’s functioning across settings and situations Understand age-expected child development Understand the content of the three child outcomes Know how to use the rating scale Understand age expectations for child functioning within the child’s culture Early Childhood Outcomes Center

Important point It is not necessary that all team members be knowledgeable in all 5 areas Especially, no expectation that parents understand the rating scale or typical child development But the professionals have to! Early Childhood Outcomes Center

Essential Knowledge for Completing the Child Outcomes Summary Form Between them, team members must: Know about the child’s functioning across settings and situations Understand age-expected child development Understand the content of the three child outcomes Know how to use the rating scale Understand age expectations for child functioning within the child’s culture Early Childhood Outcomes Center

1. Know about the child’s functioning across settings and situations How we learn about the child’s functioning across settings and situations: Good assessment Early Childhood Outcomes Center

DEC* recommended practices for assessment Involve multiple sources Examples: family members, professional team members, service providers, caregivers Involve multiple measures Examples: observations, criterion- or curriculum-based instruments, interviews, norm-referenced scales, informed clinical opinion, work samples *Division for Early Childhood Early Childhood Outcomes Center

Assessment practices appropriate for outcomes measurement: ASHA* ASHA recommended practices: Gather information from families, teachers, other service providers Collect child-centered, contextualized, descriptive, functional information (*American Speech-Language-Hearing Association) A balanced assessment, according to ASHA, includes information from families, teachers, other service providers, and information that is child-centered, contextualized, descriptive and functional. Early Childhood Outcomes Center

Assessment instruments Assessment the tool vs. assessment the process Assessment tools can inform us about children’s functioning in each of the three outcome areas Challenge: There is no assessment tool that assesses the three outcomes directly Early Childhood Outcomes Center

Essential Knowledge for Completing the Child Outcomes Summary Form Between them, team members must: Know about the child’s functioning across settings and situations Understand age-expected child development Understand the content of the three child outcomes Know how to use the rating scale Understand age expectations for child functioning within the child’s culture Early Childhood Outcomes Center

Resources for understanding age-expected child development ECO link http://www.fpg.unc.edu/~eco/pdfs/Age-expected_child_dev_9-5-07.pdf (under “ECO Tools”) New course coming soon Watch ECO web site www.the-eco-center.org Early Childhood Outcomes Center

Essential Knowledge for Completing the Child Outcomes Summary Form Between them, team members must: Know about the child’s functioning across settings and situations Understand age-expected child development Understand the content of the three child outcomes Know how to use the rating scale Understand age expectations for child functioning within the child’s culture Early Childhood Outcomes Center

Outcomes Jeopardy Biting $100 $100 $100 $200 $200 $200 $300 $300 $300 Pointing to the cabinet for cereal Reading the letter “S” on the Stop sign Washes hands before lunch Biting Plays by himself in the classroom Plays with rhyming words Building a castle from blocks with a friend Problems sleeping Sharing a cookie at lunchtime $100 $100 $100 $200 $200 $200 $300 $300 $300 Early Childhood Outcomes Center

Children have positive social relationships Involves: Relating with adults Relating with other children For older children, following rules related to groups or interacting with others Includes areas like: Attachment/separation/autonomy Expressing emotions and feelings Learning rules and expectations Social interactions and play Early Childhood Outcomes Center

Children acquire and use knowledge and skills Involves Thinking Reasoning Remembering Problem solving Using symbols and language Understanding physical and social worlds Includes: Early concepts—symbols, pictures, numbers Imitation Object permanence Expressive language and communication Early literacy Early Childhood Outcomes Center

Children take appropriate action to meet their needs Involves: Taking care of basic needs Getting from place to place Using tools (e.g., fork, toothbrush, crayon) In older children, contributing to their own health and safety Includes: Integrating motor skills to complete tasks Self-help skills (e.g., dressing, feeding, grooming, toileting, household responsibility) Acting on the world to get what one wants Early Childhood Outcomes Center

Essential Knowledge for Completing the Child Outcomes Summary Form Between them, team members must: Know about the child’s functioning across settings and situations Understand age-expected child development Understand the content of the three child outcomes Know how to use the rating scale Understand age expectations for child functioning within the child’s culture Early Childhood Outcomes Center

The two COSF questions a. To what extent does this child show age-appropriate functioning, across a variety of settings and situations, on this outcome? (Rating: 1-7) b. Has the child shown any new skills or behaviors related to [this outcome] since the last outcomes summary? (Yes-No) Early Childhood Outcomes Center

7 – Completely The child shows behaviors and skills expected in all or almost all everyday situations that are part of the child’s life Home, store, park, child care, with strangers, etc. The child’s functioning is considered appropriate for his/her age No one has significant concerns about the child’s functioning in this outcome area Early Childhood Outcomes Center

6 – Between completely and somewhat The child’s functioning generally is considered appropriate for his or her age but there are some significant concerns about the child’s functioning in this outcome area Early Childhood Outcomes Center

5 – Somewhat The child shows functioning expected for his/her age some of the time and/or in some situations The child’s functioning is a mix of age-appropriate and not appropriate functioning The child’s functioning might be described as like that of a slightly younger child We use the terminology that behaviors MIGHT BE more like those of younger children intentionally. For example, a child with autism does not display typical behaviors, but the behaviors observed would not be described as those of a younger child either. Early Childhood Outcomes Center

4 – Between a 5 and a 3 Child shows some age appropriate functioning some of the time or in some situations or settings but most of the child’s functioning would be described as not yet age appropriate. The child’s functioning might be described as like that of a younger child Early Childhood Outcomes Center

3 – Emerging The child does not yet show functioning expected of a child his/her age in any situation The child’s behaviors and skills include immediate foundational skills on which to build age-appropriate functioning The child’s functioning might be described as like that of a younger child Early Childhood Outcomes Center

2 – Between 3 and 1 The child does not yet show functioning expected of a child his/her age in any situation The child’s behaviors and skills does have some the immediate foundational skills on which to build age-appropriate functioning but these are not displayed very often The child’s functioning might be described as like that of a younger or even much younger child Early Childhood Outcomes Center

1 – Not yet The child does not yet show functioning expected of a child his/her age in any situation The child’s skills and behaviors also do not yet include any immediate foundational skills on which to build age-appropriate functioning The child’s functioning might be described as like that of a much younger child Children with 1 ratings still have skills, just not yet at an immediate foundational level Early Childhood Outcomes Center

Rating Scale Jeopardy $100 $100 $100 $200 $200 $200 $300 $300 $300 Age appropriate functioning – no concerns Mix of age appropriate and not age appropriate functioning No age appropriate functioning – not yet showing immediate foundational skills Some age appropriate functioning but very little No age appropriate functioning – lots of immediate foundational skills Age appropriate functioning – some concerns Rarely shows age appropriate functioning No age appropriate functioning – some immediate foundational skills Age appropriate functioning $100 $100 $100 $200 $200 $200 $300 $300 $300 Early Childhood Outcomes Center

Early Childhood Outcomes Center

Essential Knowledge for Completing the Child Goals Outcome Form Between them, team members must: Know about the child’s functioning across settings and situations Understand age-expected child development Understand the content of the three child outcomes Know how to use the rating scale Understand age expectations for child functioning within the child’s culture Early Childhood Outcomes Center

Point of clarification Process is NOT about comparing groups of children – it IS about asking how close children are to being able to do what is expected at their age Early learning guidelines Kindergarten and access to the general curriculum Early Childhood Outcomes Center

What we are learning nationally The process of training for child outcomes data collection has uncovered other areas of significant need related to professional development. Early Childhood Outcomes Center

Providers need to know more about: Assessment How to gather assessment data to reflect functioning across settings and situations, especially how to gather child functioning information from families Understanding the results of the assessment Sharing assessment results sensitively and honestly with families Typical child development is especially important for the COSF process but is important overall for working with young children. Early Childhood Outcomes Center

Providers need to know more about: Functional outcomes What are they? How do they differ from outcomes organized around domains? What do they mean for how professionals from different disciplines operate as a team? Typical child development What are the functional expectations for children at different ages with regard to each of the 3 goal statements? Typical child development is especially important for the COSF process but is important overall for working with young children. Early Childhood Outcomes Center

Let’s meet Ava Early Childhood Outcomes Center