Presentation is loading. Please wait.

Presentation is loading. Please wait.

Early Childhood Outcomes Center1 The Fun Stuff: Using the Data.

Similar presentations


Presentation on theme: "Early Childhood Outcomes Center1 The Fun Stuff: Using the Data."— Presentation transcript:

1 Early Childhood Outcomes Center1 The Fun Stuff: Using the Data

2 Early Childhood Outcomes Center2 Using data for program improvement = EIA E vidence I nference A ction

3 Early Childhood Outcomes Center3 Evidence Evidence refers to the numbers, such as 45% of children in category b The numbers are not debatable

4 Early Childhood Outcomes Center4 Inference How do you interpret the #s? What can you conclude from the #s? Does evidence mean good news? Bad news? News we cant interpret? To reach an inference, sometimes we analyze data in other ways (ask for more evidence)

5 Early Childhood Outcomes Center5 Inference Inference is debatable -- even reasonable people can reach different conclusions from the same set of numbers Stakeholder involvement can be helpful in making sense of the evidence

6 Early Childhood Outcomes Center6 Action Given the inference from the numbers, what should be done? Recommendations or action steps Action can be debatable – and often is Another role for stakeholders

7 Early Childhood Outcomes Center7 What can we infer? Poll results A: Candidate I.M. Good 51%, Candidate R.U. Kidding 49% (+ or – 3%) Poll results B: Candidate I.M. Good 56%, Candidate R.U. Kidding 44% (+ or – 3%)

8 Early Childhood Outcomes Center8 Program improvement: Where and how At the state level – TA, policy At the regional or local level – supervision, guidance Classroom level -- spend more time on certain aspects of the curriculum Child level -- modify intervention

9 Early Childhood Outcomes Center9 Key points Evidence refers to the numbers and the numbers by themselves are meaningless Inference is attached by those who read (interpret) the numbers You have the opportunity and obligation to attach meaning

10 Early Childhood Outcomes Center10 E – I – A Jeopardy COSF users unaware of the need to answer the yes/no progress question 90% of exit COSFs in Program B missing a response to the yes/no progress question Revise COSF procedures to emphasize completion of yes/no progress question Conduct staff development on using the 7-point rating scale 75% of children in Program A received entry ratings of 2 COSF users misunderstand the definition of points on the 7-point scale Currently used tools are not accurately assessing childrens social emotional skills Invest resources in materials for assessing social- emotional skills 45% of children reported in category e for statewide progress data, Outcome 1 $100 $200 $100 $300 $200 $300 $200 $100 $300

11 Early Childhood Outcomes Center11 Use of Data Use of Data Activity Evidence-Inference- Action

12 Early Childhood Outcomes Center12 Continuous Program Improvement Plan (vision) Program characteristics Child and family outcomes Implement Check (Collect and analyze data) Reflect Are we where we want to be?

13 Early Childhood Outcomes Center13 Tweaking the System Plan (vision) Program characteristics Child and family outcomes Implement Check (Collect and analyze data) Reflect Are we where we want to be? Is there a problem? Why is it happening? What should be done? Is it being done? Is it working?

14 Early Childhood Outcomes Center14 Continuous means… ….the cycle never ends.

15 Early Childhood Outcomes Center15 Outcome questions for program improvement, e.g. Who has good outcomes = Do outcomes vary by Region of the state? Level of functioning at entry? Services received? Age at entry to service? Type of services received? Family outcomes? Education level of parent?

16 Early Childhood Outcomes Center16 Examples of process questions Are ALL services high quality? Are ALL children and families receiving ALL the services they should in a timely manner? Are ALL families being supported in being involved in their childs program? What are the barriers to high quality services?

17 Early Childhood Outcomes Center17 Working Assumptions There are some high quality services and programs being provided across the state. There are some children who are not getting the highest quality services. If we can find ways to improve those services/programs, these children will experience better outcomes.

18 Early Childhood Outcomes Center18 Numbers as a tool Heard on the street Why are we reducing children to a number? So why do we need numbers?

19 Early Childhood Outcomes Center19

20 Early Childhood Outcomes Center20

21 Early Childhood Outcomes Center21

22 Early Childhood Outcomes Center22 +35,000

23 Early Childhood Outcomes Center23 +700,000

24 Early Childhood Outcomes Center24 Examining COSF data at one time point One group - Frequency Distribution Tables Graphs Comparing Groups Graphs Averages

25 Early Childhood Outcomes Center25 Distribution of COSF Ratings in Fall Outcome 1 Rating N% 735070 611022 5204 481.6 361.2 24.8 12.4 We are using fake data for illustration

26 Early Childhood Outcomes Center26 Frequency on Outcome 1 - Fall

27 Early Childhood Outcomes Center27 Frequency on Outcome 1 - Fall

28 Early Childhood Outcomes Center28 Comparison of two classes - Fall

29 Early Childhood Outcomes Center29 Frequency on Outcome 1 - Fall

30 Early Childhood Outcomes Center30 Frequency on Outcome 1 – Class 1

31 Early Childhood Outcomes Center31 Average Scores on Outcomes by Class – Fall, 2008 Class Social- Emotional Knowledge and Skills Action to Meet Needs 14.54.64.7 25.35.24.7 34.9 46.45.96.6 55.34.34.9 63.82.93.9 All Classes5.034.634.95

32 Early Childhood Outcomes Center32 Average Scores on Outcomes by Class – Fall, 2008 Class Social- Emotional Knowledge and Skills Action to Meet Needs 14.54.64.7 25.35.24.7 34.9 46.45.96.6 55.34.34.9 63.82.93.9 All Classes5.034.634.95

33 Early Childhood Outcomes Center33 Average Scores on Outcomes by Class – Fall, 2008 Class Social- Emotional Knowledge and Skills Action to Meet Needs 14.54.64.7 25.35.24.7 34.9 46.45.96.6 55.34.34.9 63.82.93.9 All Classes5.034.634.95

34 Early Childhood Outcomes Center34 Looking at change over time Extent of change on rating scale The OSEP categories Developmental trajectories Maintaining Changing

35 Early Childhood Outcomes Center35 Extent of change on rating scale: Time 1 to Time 2 Outcome 1 ProgressN% Maintained age-expected functioning 35070 Maintained same level function, but not age-expected 6012 Gained 3 steps102 Gained 2 steps255 Gained 1 step5010 Dropped 1 step4.8 Dropped 2 steps1.2

36 Early Childhood Outcomes Center36 13 FunctioningFunctioning

37 Early Childhood Outcomes Center37 Entry 14

38 Early Childhood Outcomes Center38 EntryExit 15

39 Early Childhood Outcomes Center39 EntryExit 16

40 Early Childhood Outcomes Center40 Key Point The OSEP categories describe types of progress children can make between entry and exit Two COSF ratings (entry and exit) are needed to calculate what OSEP category describes a child progress

41 Early Childhood Outcomes Center41 How changes in ratings on the COSF correspond to reporting categories a - e e. % of children who maintain functioning at a level comparable to same-aged peers Rated 6 or 7 at entry; AND Rated 6 or 7 at entry; AND Rated 6 or 7 at exit Rated 6 or 7 at exit

42 Early Childhood Outcomes Center42 EntryExit

43 Early Childhood Outcomes Center43 EntryExit

44 Early Childhood Outcomes Center44 EntryExit

45 Early Childhood Outcomes Center45 How changes in ratings on the COSF correspond to reporting categories a - e d. % of children who improve functioning to reach a level comparable to same-aged peers Rated 5 or lower at entry; AND Rated 5 or lower at entry; AND Rated 6 or 7 at exit Rated 6 or 7 at exit

46 Early Childhood Outcomes Center46 EntryExit

47 Early Childhood Outcomes Center47 How changes in ratings on the COSF correspond to reporting categories a - e c. % of children who improved functioning to a level nearer to same aged peers, but did not reach it Rated higher at exit than entry; AND Rated higher at exit than entry; AND Rated 5 or below at exit Rated 5 or below at exit

48 Early Childhood Outcomes Center48 EntryExit

49 Early Childhood Outcomes Center49 EntryExit

50 Early Childhood Outcomes Center50 How changes in ratings on the COSF correspond to reporting categories a - e b. % of children who improved functioning, but not sufficient to move nearer to same aged peers Rated 5 or lower at entry; AND Rated 5 or lower at entry; AND Rated the same or lower at exit; AND Rated the same or lower at exit; AND Yes on the progress question (b) Yes on the progress question (b)

51 Early Childhood Outcomes Center51 EntryExit

52 Early Childhood Outcomes Center52 EntryExit

53 Early Childhood Outcomes Center53 EntryExit

54 Early Childhood Outcomes Center54 EntryExit

55 Early Childhood Outcomes Center55 How changes in ratings on the COSF correspond to reporting categories a - e a. % of children who did not improve functioning Rated lower at exit than entry; OR Rated lower at exit than entry; OR Rated 1 at both entry and exit; AND Rated 1 at both entry and exit; AND Scored No on the progress question (b) Scored No on the progress question (b)

56 Early Childhood Outcomes Center56 EntryExit

57 Early Childhood Outcomes Center57 EntryExit

58 Early Childhood Outcomes Center58 Federal Reporting Categories Activity COSF Ratings to Progress Categories

59 ECO Calculator

60 Early Childhood Outcomes Center60 ECO Calculator (Tutor) http://www.fpg.unc.edu/~eco/outcomes.cfm

61 Early Childhood Outcomes Center61 OSEP progress categories Looking at information across time Reducing the information to fewer categories to allow easier comparisons

62 OSEP Categories 2009 (%) 2010 (%) 2011 (%) Maintained Age Appro Trajec 232224 Changed Traj – Age Appro 151713 Changed Traj – Closer to Age App 323437 Same Trajectory -Progress 2825 Flat Trajectory – No Prog. 221

63 OSEP Categories 2009 (%) 2010 (%) 2011 (%) Maintained Age Appro Trajec 232224 Changed Traj – Age Appro 151713 TOTAL - Age Appropriate at Exit 383937

64 OSEP Categories Class 1 (%) Class 2 (%) Class 3 (%) Maintained Age Appro Trajec 232224 Changed Traj – Age Appro 151713 TOTAL - Age Appropriate at Exit 383937

65 OSEP Categories 2009 (%) 2010 (%) 2011 (%) Changed Traj – Age Appro 151718 Changed Traj – Closer to Age App 323437 TOTAL – Greater than Expected Progress 475155

66 Early Childhood Outcomes Center66 Working with data Different levels of analysis are required for different levels of questions Aggregation will work for you – but loses detail about individual children. 50 assessment items on 20 children in 5 classes in Fall and Spring 50 x 20 x 5 x 2 = 10,000 pieces of information

67 Early Childhood Outcomes Center67 Using assessment data at the classroom level Looking at the data by child At a single point in time Over time Looking at data for areas that cut across children At a single point in time Over time

68 Name Items Related to Outcome 1 1 Plays well with others 2 Cooperates with peers in simple games 3 Stops for transition cues 4 Takes directions well from adults 5 Has at least one close friend CarlosAEEAA GeetaNY EileenAAAAA MingEEENY ShaniquaEEEEA Example: Item Results for 5 Imaginary Children A=Accomplished; E= Emerging; NY= Not yet

69 Example: COSF Outcome Ratings for Class 3c by Child Name Outcome 1Outcome 2Outcome 3 Time 1 Time 2 Time 1 Time 2 Time 1 Time 2 Carlos 563455 Geeta 122234 Eileen 777777 Ming 224423 Shaniqua 343455

70 Example of an Aggregated Report for Program: Percentage of Children Scoring 5 or Higher on COSF by Class Class Outcome 1Outcome 2Outcome 3 Time 1Time 2Time 1Time 2Time 1Time 2 1a657050514952 1b555362618788 2a474351676566 2b768478857883 3a9798959795100 What do you see in these data?

71 Early Childhood Outcomes Center71 Outcome questions for program improvement, e.g. Who has good outcomes = Do outcomes vary by Region of the state? Level of functioning at entry? Services received? Age at entry to service? Type of services received? Family outcomes? Education level of parent?

72 Early Childhood Outcomes Center72 Looking at Data by Region Class 1Class 2Class 3 454723 Percentage of Children Who Changed Developmental Trajectories After One Year of Service Possible inference?

73 Early Childhood Outcomes Center73 Looking at Data by Age at Entry 36 to 40 months 41 to 44 months 45 to 49 months 344246 Percentage of Children Who Changed Developmental Trajectories After One Year of Service Possible inference?

74 Early Childhood Outcomes Center74 Take Home Message You will want to look at your data in lots of different ways You will want to think about the possible inferences You may need other information to decide among possible inferences Act on what you have learned

75 Early Childhood Outcomes Center75 Tweaking the System Plan (vision) Program characteristics Child and family outcomes Implement Check (Collect and analyze data) Reflect Are we where we want to be? Is there a problem? Why is it happening? What should be done? Is it being done? Is it working?

76 Early Childhood Outcomes Center76 How will/might these data be used? Federal level Overall funding decisions (accountability) Resource allocation (e.g., what kind of TA to fund?) Decisions about effectiveness of program in individual states State level Program effectiveness?? Program improvement?? Local level Program improvement??

77 Early Childhood Outcomes Center77 Need for good data Encompasses all three levels: federal, state, local Depends on how well local programs are implementing procedures

78 Early Childhood Outcomes Center78 Many steps to ensuring quality data Before Good data collection/Training Good data system and data entry procedures During Ongoing supervision of implementation Feedback to implementers Refresher training After Review of COSF records Data analyses for validity checks

79 Early Childhood Outcomes Center79 Take Home Message If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else. Inference = Data not yet valid Action = Continue to improve data collection and quality assurance

80 Early Childhood Outcomes Center80 Data Exploration Examine the data to look for inconsistencies If and when you find something strange, look for some other data you have that might help explain it. Is the variation caused by something other than bad data?

81 Early Childhood Outcomes Center81 Obtaining good data Focus on addressing the threats to good data Local providers do not understand the procedures Local providers do not follow the procedures And others….. Identify and address the threats

82 Early Childhood Outcomes Center82 How far along is Illinois?

83 Early Childhood Outcomes Center83 Keeping our eye on the prize: High quality services for children and families that will lead to good outcomes.

84 Early Childhood Outcomes Center84 For more information…. www.the-eco-center.org


Download ppt "Early Childhood Outcomes Center1 The Fun Stuff: Using the Data."

Similar presentations


Ads by Google