1. Accountability: Getting it Right in Early Childhood Kathy Hebbeler Early Childhood Outcomes Center at SRI International Presented at the Measuring Child and Family Outcomes Meeting, Baltimore 2008

2. Early Childhood Outcomes Center2 Where are you?

3. Early Childhood Outcomes Center3 Today we hope to… Begin a conversation about what an ideal EC accountability system would look like Describe some recent developments in EC Accountability Support you as state leader in shaping the EC accountability system in your state.

4. Early Childhood Outcomes Center4 What is happening out there? Incredible expansion of EC programming at the state level Accountability is not going away How best to do accountability is still a topic for hot conversation in early childhood Essential that persons knowledgeable about IDEA programs be part of those conversations

5. Early Childhood Outcomes Center5 For what do we want to be held accountable? a)High quality services provided through 619 and Part C b)Good outcomes for children with special needs and their families c)Both

6. Early Childhood Outcomes Center6 And who are “we?” a)Administrators and others associated with Part C and Part B 619 programs? b)Advocates concerned about how young children in our society are faring? c)Both

7. Early Childhood Outcomes Center7 Good Accountability requires Intentionality Being proactive Having a vision So what is your state’s vision for your accountability system?

8. Early Childhood Outcomes Center8 Good Accountability requires.. Seeing accountability as a system …. …A group of interacting, interrelated, or interdependent elements forming a complex whole

9. Early Childhood Outcomes Center9 EC System (or not) State Pre-K Public Health home visiting Child Care Part C 619 Early Head Start Head Start

10. Early Childhood Outcomes Center10 System Under Construction State Pre-K Child Care Part C Head Start Transition Accountability 619

11. Early Childhood Outcomes Center11 Components of IDEA Service System Service Delivery Child Find System entry – eligibility determination Service provision Transition Etc. Supports to Service Delivery Policy, legislation, regulation Professional Development Accountability Etc.

12. Early Childhood Outcomes Center12 The Accountability System is a component within the Service System System within a system

13. Early Childhood Outcomes Center13 Two Recent Reports

14. Early Childhood Outcomes Center14 State Early Childhood Accountability and Improvement System A system of standards-based assessments of (a) children’s development and learning and (b) program quality, designed to inform state policy decisions, investments, and improvement efforts for early education programs for three- and four-year-old children, linked to a continuum of kindergarten through third grade standards, curriculum, assessments, and program improvement efforts. National Early Childhood Accountability Task Force

15. Early Childhood Outcomes Center15 A system of standards-based assessments of (a) children’s development and learning and (b) program quality, designed to inform state policy decisions, investments, and improvement efforts for early education programs for three- and four-year-old children, linked to a continuum of kindergarten through third grade standards, curriculum, assessments, and program improvement efforts. State Early Childhood Accountability and Improvement System National Early Childhood Accountability Task Force Child Outcomes Linkage Purpose Program information

16. Early Childhood Outcomes Center16 Three Primary Building Blocks System Infrastructure Assessment/Program Improvement Approaches Steps toward Coherent PreK-Grade 3 Accountability Efforts National Early Childhood Accountability Task Force

17. Early Childhood Outcomes Center17 Infrastructure ….An underlying base or foundation especially for an organization or system ….The basic facilities, services, and installations needed for the functioning of a community or society National Early Childhood Accountability Task Force

18. Early Childhood Outcomes Center18 Four Parts of the EC Accountability Infrastructure Early learning and program quality standards Program rating and improvement system Professional development system Data management and reporting system National Early Childhood Accountability Task Force

19. Early Childhood Outcomes Center19 Assessment: Importance of Purpose Make purpose of assessment explicit and public Match the assessment strategy to the purpose Administrators, teachers, and assessors should be able to articulate the purpose of an assessment NRC Report - Early Childhood Assessment: Why, What, and How?

20. Early Childhood Outcomes Center20 Assessment system is part of EC larger system Standards Assessments Reporting (data base) Professional development related to standards and assessment Procedures to assess opportunity to learn (Program quality) Fair assessment for all children Sufficient resources to implement Ongoing monitoring and evaluation NRC Report - Early Childhood Assessment: Why, What, and How?

21. Early Childhood Outcomes Center21 Relationship between these two reports Task Force Accountability Report – Focus: EC Accountability Assessment is a component of an accountability system NRC Report on Assessment Focus: EC Assessment Accountability is one purpose for assessment

22. Early Childhood Outcomes Center22 Components of an Accountability System Vision (related to Vision for Service System) Purpose

23. Early Childhood Outcomes Center23 Articulating purpose for state Accountability System Probably goes beyond “Reporting data for the SPP/APR” Needs to be more specific than “accountability and program improvement” Note: Purpose can vary with level; OSEP can have different purposes from state’s purpose(s) as can local programs.

24. Early Childhood Outcomes Center24 Possible purposes  Demonstrate effectiveness of program for policy- makers (governor, legislature)  Identify local programs in need of improvement  Determine whether services are equally effective for different sub-groups, e.g.,  Children with health problems  Low income families, etc.  Determine whether services are equally effective across different outcome areas.  _______________________________________ Check all that apply

25. Early Childhood Outcomes Center25 Communication is key Purpose(s) needs to be Clearly articulated In writing Readily available Widely disseminated No searching. No secrets. No suspicions.

26. Early Childhood Outcomes Center26 How to Achieve Purpose(s)? 1.Build evidence base so state can reach valid conclusions about service system. 2.Use evidence base to accomplish stated purposes.

27. Early Childhood Outcomes Center27 Build the evidence base FMA* What does state want to know about what? *Findings, Meaning, Action

28. Early Childhood Outcomes Center28 Evidence about what? Accountability Task Force: A system of standards-based assessments of (a) children’s development and learning and (b) program quality Note: This assumes the service system has standards for children’s development and learning and standards for program quality so they can be assessed

29. Early Childhood Outcomes Center29 Standards for Children’s Learning and Development: Early Learning Guidelines Knowledge, skills, habits, attitudes, dispositions children are expected to acquire by a given point in time Almost every state has them for 3-5 year olds Many states already have them for 0-3; others with them under development.

31. Early Childhood Outcomes Center31

32. Early Childhood Outcomes Center32 Early Learning Guidelines Developed for multiple purpose Considerable variations in content, format, specificity across states Stand as public statements of what we want children to do at various ages Public policy statement of developmental expectations

33. Early Childhood Outcomes Center33

34. Early Childhood Outcomes Center34

35. Early Childhood Outcomes Center35 What we know (WWK) To paraphrase Thomas Jefferson, “We hold these truths to be evidence-based….

36. Early Childhood Outcomes Center36 WWK: Children’s development proceeds according to a general timeline Competencies are acquired within an age range; developmental milestones Some skills precede other skills Developmental timeline is the basis for screening and assessment tools in early childhood; also provides foundation for ELGs

37. Early Childhood Outcomes Center37 WWK: Current developmental status predicts future developmental status Current functioning is a statement about where a child is Also, a prediction or odds for where the child will be in the future. Basis for developmental trajectories Goal of intervention is to change the odds.

39. Early Childhood Outcomes Center39 WWK: There is a lot of variation in children’s development/learning but some children’s development/learning is problematic Some is “just variation” Some is troublesome – because it bodes ill for the child’s future (bad odds) Critical that we can distinguish Policy response: Early intervention and early childhood special education State criteria for delay – the point at which the delay is so substantial, the child is not likely to “outgrow” it. Eligibility for IDEA services.

40. Early Childhood Outcomes Center40 Thinking About Age Expectations Age- expected skills and behavior Movement away from age-expected Movement toward age- expected

41. Inequality at ECLS-K data from Inequality at the Starting Gate

42. inequality

43. Early Childhood Outcomes Center43 WWK: There is a lot of variation in children’s development/learning but some children’s development/learning is problematic Policy response: Head Start State Pre-K Child Care Quality Standards Goal: Alter the trajectory. Change the odds

44. Early Childhood Outcomes Center44 WWK: There is a lot of variation in children’s development/learning but some children’s development/learning is problematic Policy response: Early learning guidelines [Hopefully] Grounded in what we know about development, ELGs lay out developmental expectations Provide guidance to keep children “on track” or get them back on track so they arrive at kindergarten with a certain set of skills

45. Early Childhood Outcomes Center45 Understanding the difference between odds and a sure thing Need to understand how to interpret data on children’s current functioning Not wise to obsess on slight degrees of variation Not wise to ignore substantial deviation (“grow out of it”) A child “off track” is a prediction begging to be proven wrong – if effective intervention is provided (WWK: Early intervention works.)

46. Early Childhood Outcomes Center46 Application of ELGs to children with special needs Do ELGs apply? Are they useful? At a individual child or classroom level? At a state level? They are statements of what is expected at given ages Parallels with the application of standards for children with disabilities in K-12? Have fought for students’ rights to be held (and compared) to the same standards as all students Standards provide access to higher expectations for students with disabilities Could ELGs be a common currency in moving toward unified EC system?

47. Early Childhood Outcomes Center47 Application of ELGs to children with special needs DEC Recommended Practice: A41. Professionals monitor child progress based on past performance as the referent rather than on group norms. This RP is speaking to norm-referenced tests. Implication is that the only thing we want to look at is child compared to him’/herself. ELGs are not group norms – but they are group standards. Can ELGs provide directions to Part C and 619 programs for age expectations?

48. Early Childhood Outcomes Center48 Lessons ECO has learned EI and 619 providers need to know more about typical child development And so do child care providers, etc… Some lack of clarity over how to determine age expectations for OSEP reporting purposes Provider discomfort over how to talk to parents about current level of functioning when it is low

49. Early Childhood Outcomes Center49 PA’s red flags “…help adults gain a sense of whether additional screening or evaluation should be sought if a particular skill has not been acquired within a specific developmental age span.” Help adults watch for possible delays.

50. Early Childhood Outcomes Center50 What kind of evidence about children’s learning and development? Task Force Recommendation: Measure progress and status. OSEP reporting does both! Checks on status at 3 years and 5 years. Looks at progress over time in program.

51. Early Childhood Outcomes Center51 Evidence about what? Accountability Task Force: A system of standards-based assessments of (a) children’s development and learning and (b) program quality Recommended expansion: Include family outcomes.

52. Early Childhood Outcomes Center52 What do we want to know about programs? Both ATF and NRC reports speak to looking at program quality Possible addition: Reframe to be program characteristics and program quality Program characteristics: Hours, months of operation Program model Services offered Curriculum Program quality measures

53. Early Childhood Outcomes Center53 What do we want to know about programs? Important difference in “program” for C/619 compared to general EC Some program measures apply to a group of children (e.g., measures of classroom quality) Some are individualized (e.g., receipt of speech therapy) Important implications for measurement

54. Early Childhood Outcomes Center54 What do we want to know about programs? What is the child’s program? Services Minutes of services Location of service provision What is the quality of the child’s program? How should we assess quality, especially of the individualized component?

55. Early Childhood Outcomes Center55 Program quality measures Examples: Early Childhood Rating Scale (ECERS), Early Language and Literacy Classroom Observation (ELLCO), Classroom Assessment Scoring System (CLASS) See NRC report for a comprehensive review of program quality tools

56. Early Childhood Outcomes Center56 Evidence about what? Possible Modifications to ATF Recommendations Child and family outcomes Program (Group, individual) Characteristics Quality Infrastructure supporting the program Legislation policies Professional development, etc.

57. System for Producing Good Child and Family Outcomes High quality services and supports for children 0-5 and their families Good outcomes for children and families

58. Prof’l Development Preservice Inservice System for Producing Good Child and Family Outcomes Good Federal policies and programs Good State policies and programs High quality services and supports for children 0-5 and their families Good outcomes for children and families Good Local policies and programs Adequate funding Strong Leadership

59. Early Childhood Outcomes Center59 Ecological Model of Child Development Child Family

60. Early Childhood Outcomes Center60 Ecological Model of Child Development Child Family Community

61. Early Childhood Outcomes Center61 Ecological Model of Program Implementation Multiple forces impact how programs are implemented Important to know how your programs look Also, important to know what leads them to look like this Forces acting on programs provide levers for program improvement

62. Early Childhood Outcomes Center62 Concluding Caveat NEVER ever reach a conclusion about program effectiveness based on child outcome data alone Best practice (at all levels of decision- making): Multiple sources of information Conclusion: Accountability system needs evidence about outcomes AND evidence about program

63. Early Childhood Outcomes Center63 Components of an Accountability System Vision (related to Vision for Service System) Purpose Evidence Base

64. Early Childhood Outcomes Center64 How to get the evidence 1.Good measurement procedures = procedures capable of producing high quality date Good system design 2.Fidelity of implementation = procedures are implemented as intended Good quality assurance

65. Early Childhood Outcomes Center65 Components of an Accountability System Vision (related to Vision for Service System) Purpose Evidence Base Use of evidence to accomplish purposes

66. Early Childhood Outcomes Center66 Use of evidence* Reaching valid conclusions (M) Taking effective action (A) *Addressing the purposes

67. Early Childhood Outcomes Center67 Taking the fear out of accountability What are we afraid of? Findings/Evidence Meaning/Conclusions Action Continuum of actions What are the options on the continuum?

68. Early Childhood Outcomes Center68 Accountability Nightmare Meaning – reaching the wrong conclusion Action – leading to actions that will ultimately harm children and families

69. Early Childhood Outcomes Center69 Avoiding the nightmare Type of wrong conclusion False negative –program is ineffective when it is effective Overall For some types of children In some areas False positive – program is effective when it isn’t

70. Early Childhood Outcomes Center70 Alignment of Conclusion and Reality EffectiveIneffective EffectiveCorrect! False Positive Ineffective False Negative Correct! Reality Conclusion

71. Early Childhood Outcomes Center71 Avoiding the nightmare Good evidence Credible Multiple pieces of evidence Evidence about child outcomes Evidence about program factors (quality)* *See Accountability Task Force

72. Early Childhood Outcomes Center72 Need for outcomes AND program data Avoid both false negatives and positives by looking at multiple pieces of evidence Triangulation Do multiple sources of evidence point to the same conclusion? The outcomes data; the program data; data on who is being served, etc.

73. Early Childhood Outcomes Center73 Avoiding the nightmare: Solid Infrastructure for an Accountability System Clear vision Clear articulation of purpose and public statement as to how the data will be used Credible evidence (drill down here…)

74. Early Childhood Outcomes Center74 Clear statement of possible future actions What will be the consequences of a negative finding? Sanctions?? The ultimate worst nightmare is the prospect of a false negative combined with sanctions Never consider sanctions based on child outcomes data alone (or any single piece of evidence)

75. Early Childhood Outcomes Center75 Clear statement of possible actions Supports to improve weak programs or weak program areas Site visits Individualized technical assistance Professional development plan Etc. Note: If the wrong program is identified (false negative) and provided resources, no harm comes to children and families in that program but it is a waste of resources.

76. Early Childhood Outcomes Center76 Clear statement of possible actions False positives Sort of where we are now: programs, investments assumed effective..but some of them probably aren’t Consequences Short changes children and families Some children and families will not experience the kinds of outcomes they could have with a quality program.

77. Early Childhood Outcomes Center77 Sidebar thought Why are we more worried about false negatives than false positives? It would be damaging to children and families to defund an effective program but it also is damaging to have children and families receiving ineffective services….

78. Early Childhood Outcomes Center78 Sidebar thought #2 What about comparing states [programs]? What kinds of conclusions will OSEP reach? What are the actions/consequences that will follow? OSEP has no intention of comparing states on outcomes

79. Early Childhood Outcomes Center79 What about comparing? Can’t afford to be naïve Once data are out there, anyone can make comparisons (state to state; within state, local to local) Comparing is not the problem; invalid conclusions (especially false negatives) are the problem Using comparisons to identify weak programs is what accountability and program improvement is supposed to do

80. Early Childhood Outcomes Center80 Best protection against false negatives (and false positives) Good evidence base Multiple sources of information on: Children and families Programs Infrastructure Credible, well implemented measurement strategies

81. Early Childhood Outcomes Center81 Putting Good Accountability to Work 1.Build the infrastructure for producing a body of evidence that will allow valid conclusions 2.Compile the evidence and carefully reach conclusions 3.Act on those conclusions to build a better system of services 4.Repeat 2 and 3 indefinitely.

82. Early Childhood Outcomes Center82 Good EC Systems Build on what we know (WWK) About children’s learning and development Multiple influences on development About adult learning About innovation, implementation Etc. (this is another way of saying they are evidenced-based) To provide high quality services and supports

83. Early Childhood Outcomes Center83 Accountability Task Force Recommendation States should develop a single unified and coherent system of standards, assessments, data and professional development efforts across early childhood categorical programs and funding streams. ** Alignment ** ** Systems Integration **

84. Early Childhood Outcomes Center84 Turning an EC Non-System into a System What resources exist in your state to promote good outcomes for children birth to 5? Which of these resources work together? How? Share a vision of common outcomes? Build a seamless system for families? Joint professional development? Shared vision for transition? Unified accountability system?

85. Early Childhood Outcomes Center85 Producing Good Outcomes for Young Children with Disabilities Proposition: Good IDEA services are not enough to effectively change the odds for some(?)/many(?) young children with special needs. Need the entire EC System to be functioning effectively.

86. Early Childhood Outcomes Center86 Producing Good Outcomes for Young Children with Disabilities Need more forces at work to change the odds and help children meet developmental expectations. Need good child care (and an accountability system) Need good PreK and Head Start programs

87. Early Childhood Outcomes Center87 Producing Good Outcomes for Young Children with Disabilities Need the EC services to work together Need an accountability system that can look at the entire EC system

88. Early Childhood Outcomes Center88 For what do we want to be held accountable? a)High quality services provided through 619 and Part C b)Good outcomes for children with special needs and their families c)Both

89. Early Childhood Outcomes Center89 And who are “we?” a)Administrators and others associated with Part C and Part B 619 programs? b)Advocates concerned about how young children in our society are faring? c)Both

90. Early Childhood Outcomes Center90 References Task Force on Accountability (2007): Taking Stock: Assessing and Improving Early Childhood Learning and Program Quality http://www.pewtrusts.org/our_work_detail.aspx?id=102 National Research Council; Snow & Van Hemel (2008): Early Childhood Assessment: What, Why, and How? http://www.nap.edu/catalog.php?record_id=12446

91. Early Childhood Outcomes Center91 References Web site on state Early Learning Guidelines http://www.ccsso.org/content/PDFs/State_ECstandardsMA TRIX11.27.07.pdf http://www.ccsso.org/content/PDFs/State_ECstandardsMA TRIX11.27.07.pdf Research on State’s ELGs –Search on “Catherine Scott Little” on Google Scholar Lee, V. E., & Burkam, D. T. (2002). Inequality at the starting gate: Social background differences in achievement as children begin school. Washington, DC: Economic Policy Institute.