Presentation on theme: "Early Childhood Outcomes Center1 The Fun Stuff: Using the Data."— Presentation transcript:
Early Childhood Outcomes Center1 The Fun Stuff: Using the Data
Early Childhood Outcomes Center2 Using data for program improvement = EIA E vidence I nference A ction
Early Childhood Outcomes Center3 Evidence Evidence refers to the numbers, such as 45% of children in category b The numbers are not debatable
Early Childhood Outcomes Center4 Inference How do you interpret the #s? What can you conclude from the #s? Does evidence mean good news? Bad news? News we cant interpret? To reach an inference, sometimes we analyze data in other ways (ask for more evidence)
Early Childhood Outcomes Center5 Inference Inference is debatable -- even reasonable people can reach different conclusions from the same set of numbers Stakeholder involvement can be helpful in making sense of the evidence
Early Childhood Outcomes Center6 Action Given the inference from the numbers, what should be done? Recommendations or action steps Action can be debatable – and often is Another role for stakeholders
Early Childhood Outcomes Center7 What can we infer? Poll results A: Candidate I.M. Good 51%, Candidate R.U. Kidding 49% (+ or – 3%) Poll results B: Candidate I.M. Good 56%, Candidate R.U. Kidding 44% (+ or – 3%)
Early Childhood Outcomes Center8 Program improvement: Where and how At the state level – TA, policy At the regional or local level – supervision, guidance Classroom level -- spend more time on certain aspects of the curriculum Child level -- modify intervention
Early Childhood Outcomes Center9 Key points Evidence refers to the numbers and the numbers by themselves are meaningless Inference is attached by those who read (interpret) the numbers You have the opportunity and obligation to attach meaning
Early Childhood Outcomes Center10 E – I – A Jeopardy COSF users unaware of the need to answer the yes/no progress question 90% of exit COSFs in Program B missing a response to the yes/no progress question Revise COSF procedures to emphasize completion of yes/no progress question Conduct staff development on using the 7-point rating scale 75% of children in Program A received entry ratings of 2 COSF users misunderstand the definition of points on the 7-point scale Currently used tools are not accurately assessing childrens social emotional skills Invest resources in materials for assessing social- emotional skills 45% of children reported in category e for statewide progress data, Outcome 1 $100 $200 $100 $300 $200 $300 $200 $100 $300
Early Childhood Outcomes Center11 Use of Data Use of Data Activity Evidence-Inference- Action
Early Childhood Outcomes Center12 Continuous Program Improvement Plan (vision) Program characteristics Child and family outcomes Implement Check (Collect and analyze data) Reflect Are we where we want to be?
Early Childhood Outcomes Center13 Tweaking the System Plan (vision) Program characteristics Child and family outcomes Implement Check (Collect and analyze data) Reflect Are we where we want to be? Is there a problem? Why is it happening? What should be done? Is it being done? Is it working?
Early Childhood Outcomes Center14 Continuous means… ….the cycle never ends.
Early Childhood Outcomes Center15 Outcome questions for program improvement, e.g. Who has good outcomes = Do outcomes vary by Region of the state? Level of functioning at entry? Services received? Age at entry to service? Type of services received? Family outcomes? Education level of parent?
Early Childhood Outcomes Center16 Examples of process questions Are ALL services high quality? Are ALL children and families receiving ALL the services they should in a timely manner? Are ALL families being supported in being involved in their childs program? What are the barriers to high quality services?
Early Childhood Outcomes Center17 Working Assumptions There are some high quality services and programs being provided across the state. There are some children who are not getting the highest quality services. If we can find ways to improve those services/programs, these children will experience better outcomes.
Early Childhood Outcomes Center18 Numbers as a tool Heard on the street Why are we reducing children to a number? So why do we need numbers?
Early Childhood Outcomes Center19
Early Childhood Outcomes Center20
Early Childhood Outcomes Center21
Early Childhood Outcomes Center22 +35,000
Early Childhood Outcomes Center ,000
Early Childhood Outcomes Center24 Examining COSF data at one time point One group - Frequency Distribution Tables Graphs Comparing Groups Graphs Averages
Early Childhood Outcomes Center25 Distribution of COSF Ratings in Fall Outcome 1 Rating N% We are using fake data for illustration
Early Childhood Outcomes Center26 Frequency on Outcome 1 - Fall
Early Childhood Outcomes Center27 Frequency on Outcome 1 - Fall
Early Childhood Outcomes Center28 Comparison of two classes - Fall
Early Childhood Outcomes Center29 Frequency on Outcome 1 - Fall
Early Childhood Outcomes Center30 Frequency on Outcome 1 – Class 1
Early Childhood Outcomes Center31 Average Scores on Outcomes by Class – Fall, 2008 Class Social- Emotional Knowledge and Skills Action to Meet Needs All Classes
Early Childhood Outcomes Center32 Average Scores on Outcomes by Class – Fall, 2008 Class Social- Emotional Knowledge and Skills Action to Meet Needs All Classes
Early Childhood Outcomes Center33 Average Scores on Outcomes by Class – Fall, 2008 Class Social- Emotional Knowledge and Skills Action to Meet Needs All Classes
Early Childhood Outcomes Center34 Looking at change over time Extent of change on rating scale The OSEP categories Developmental trajectories Maintaining Changing
Early Childhood Outcomes Center35 Extent of change on rating scale: Time 1 to Time 2 Outcome 1 ProgressN% Maintained age-expected functioning Maintained same level function, but not age-expected 6012 Gained 3 steps102 Gained 2 steps255 Gained 1 step5010 Dropped 1 step4.8 Dropped 2 steps1.2
Early Childhood Outcomes Center36 13 FunctioningFunctioning
Early Childhood Outcomes Center37 Entry 14
Early Childhood Outcomes Center38 EntryExit 15
Early Childhood Outcomes Center39 EntryExit 16
Early Childhood Outcomes Center40 Key Point The OSEP categories describe types of progress children can make between entry and exit Two COSF ratings (entry and exit) are needed to calculate what OSEP category describes a child progress
Early Childhood Outcomes Center41 How changes in ratings on the COSF correspond to reporting categories a - e e. % of children who maintain functioning at a level comparable to same-aged peers Rated 6 or 7 at entry; AND Rated 6 or 7 at entry; AND Rated 6 or 7 at exit Rated 6 or 7 at exit
Early Childhood Outcomes Center42 EntryExit
Early Childhood Outcomes Center43 EntryExit
Early Childhood Outcomes Center44 EntryExit
Early Childhood Outcomes Center45 How changes in ratings on the COSF correspond to reporting categories a - e d. % of children who improve functioning to reach a level comparable to same-aged peers Rated 5 or lower at entry; AND Rated 5 or lower at entry; AND Rated 6 or 7 at exit Rated 6 or 7 at exit
Early Childhood Outcomes Center46 EntryExit
Early Childhood Outcomes Center47 How changes in ratings on the COSF correspond to reporting categories a - e c. % of children who improved functioning to a level nearer to same aged peers, but did not reach it Rated higher at exit than entry; AND Rated higher at exit than entry; AND Rated 5 or below at exit Rated 5 or below at exit
Early Childhood Outcomes Center48 EntryExit
Early Childhood Outcomes Center49 EntryExit
Early Childhood Outcomes Center50 How changes in ratings on the COSF correspond to reporting categories a - e b. % of children who improved functioning, but not sufficient to move nearer to same aged peers Rated 5 or lower at entry; AND Rated 5 or lower at entry; AND Rated the same or lower at exit; AND Rated the same or lower at exit; AND Yes on the progress question (b) Yes on the progress question (b)
Early Childhood Outcomes Center51 EntryExit
Early Childhood Outcomes Center52 EntryExit
Early Childhood Outcomes Center53 EntryExit
Early Childhood Outcomes Center54 EntryExit
Early Childhood Outcomes Center55 How changes in ratings on the COSF correspond to reporting categories a - e a. % of children who did not improve functioning Rated lower at exit than entry; OR Rated lower at exit than entry; OR Rated 1 at both entry and exit; AND Rated 1 at both entry and exit; AND Scored No on the progress question (b) Scored No on the progress question (b)
Early Childhood Outcomes Center56 EntryExit
Early Childhood Outcomes Center57 EntryExit
Early Childhood Outcomes Center58 Federal Reporting Categories Activity COSF Ratings to Progress Categories
Early Childhood Outcomes Center60 ECO Calculator (Tutor)
Early Childhood Outcomes Center61 OSEP progress categories Looking at information across time Reducing the information to fewer categories to allow easier comparisons
OSEP Categories 2009 (%) 2010 (%) 2011 (%) Maintained Age Appro Trajec Changed Traj – Age Appro Changed Traj – Closer to Age App Same Trajectory -Progress 2825 Flat Trajectory – No Prog. 221
OSEP Categories 2009 (%) 2010 (%) 2011 (%) Maintained Age Appro Trajec Changed Traj – Age Appro TOTAL - Age Appropriate at Exit
OSEP Categories Class 1 (%) Class 2 (%) Class 3 (%) Maintained Age Appro Trajec Changed Traj – Age Appro TOTAL - Age Appropriate at Exit
OSEP Categories 2009 (%) 2010 (%) 2011 (%) Changed Traj – Age Appro Changed Traj – Closer to Age App TOTAL – Greater than Expected Progress
Early Childhood Outcomes Center66 Working with data Different levels of analysis are required for different levels of questions Aggregation will work for you – but loses detail about individual children. 50 assessment items on 20 children in 5 classes in Fall and Spring 50 x 20 x 5 x 2 = 10,000 pieces of information
Early Childhood Outcomes Center67 Using assessment data at the classroom level Looking at the data by child At a single point in time Over time Looking at data for areas that cut across children At a single point in time Over time
Name Items Related to Outcome 1 1 Plays well with others 2 Cooperates with peers in simple games 3 Stops for transition cues 4 Takes directions well from adults 5 Has at least one close friend CarlosAEEAA GeetaNY EileenAAAAA MingEEENY ShaniquaEEEEA Example: Item Results for 5 Imaginary Children A=Accomplished; E= Emerging; NY= Not yet
Example: COSF Outcome Ratings for Class 3c by Child Name Outcome 1Outcome 2Outcome 3 Time 1 Time 2 Time 1 Time 2 Time 1 Time 2 Carlos Geeta Eileen Ming Shaniqua
Example of an Aggregated Report for Program: Percentage of Children Scoring 5 or Higher on COSF by Class Class Outcome 1Outcome 2Outcome 3 Time 1Time 2Time 1Time 2Time 1Time 2 1a b a b a What do you see in these data?
Early Childhood Outcomes Center71 Outcome questions for program improvement, e.g. Who has good outcomes = Do outcomes vary by Region of the state? Level of functioning at entry? Services received? Age at entry to service? Type of services received? Family outcomes? Education level of parent?
Early Childhood Outcomes Center72 Looking at Data by Region Class 1Class 2Class Percentage of Children Who Changed Developmental Trajectories After One Year of Service Possible inference?
Early Childhood Outcomes Center73 Looking at Data by Age at Entry 36 to 40 months 41 to 44 months 45 to 49 months Percentage of Children Who Changed Developmental Trajectories After One Year of Service Possible inference?
Early Childhood Outcomes Center74 Take Home Message You will want to look at your data in lots of different ways You will want to think about the possible inferences You may need other information to decide among possible inferences Act on what you have learned
Early Childhood Outcomes Center75 Tweaking the System Plan (vision) Program characteristics Child and family outcomes Implement Check (Collect and analyze data) Reflect Are we where we want to be? Is there a problem? Why is it happening? What should be done? Is it being done? Is it working?
Early Childhood Outcomes Center76 How will/might these data be used? Federal level Overall funding decisions (accountability) Resource allocation (e.g., what kind of TA to fund?) Decisions about effectiveness of program in individual states State level Program effectiveness?? Program improvement?? Local level Program improvement??
Early Childhood Outcomes Center77 Need for good data Encompasses all three levels: federal, state, local Depends on how well local programs are implementing procedures
Early Childhood Outcomes Center78 Many steps to ensuring quality data Before Good data collection/Training Good data system and data entry procedures During Ongoing supervision of implementation Feedback to implementers Refresher training After Review of COSF records Data analyses for validity checks
Early Childhood Outcomes Center79 Take Home Message If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else. Inference = Data not yet valid Action = Continue to improve data collection and quality assurance
Early Childhood Outcomes Center80 Data Exploration Examine the data to look for inconsistencies If and when you find something strange, look for some other data you have that might help explain it. Is the variation caused by something other than bad data?
Early Childhood Outcomes Center81 Obtaining good data Focus on addressing the threats to good data Local providers do not understand the procedures Local providers do not follow the procedures And others….. Identify and address the threats
Early Childhood Outcomes Center82 How far along is Illinois?
Early Childhood Outcomes Center83 Keeping our eye on the prize: High quality services for children and families that will lead to good outcomes.
Early Childhood Outcomes Center84 For more information….