1. Head Start National Reporting System Assessment Development, First-Year Results, Future Plans Nicholas Zill, Ph.D. Westat Presented at the National Head Start Association 32 nd Annual Training Conference May 26, 2005

2. 2 Purpose & Functions The Head Start National Reporting System (NRS) was designed to address the President’s Good Start, Grow Smart initiative and 1998 Congressional mandates for program. NRS provides indicators of the progress children are making on key early literacy and math skills for all local programs.

3. 3 Purpose & Functions (2) NRS supplements local child assessment and program self-assessment efforts. For the first time, provides Head Start with consistently-collected, comparable measures for all programs. To be used in planning training and technical assistance for programs. Incorporate in future program monitoring.

4. 4 How NRS Works All kindergarten-eligible 4- and 5-year-old children are given the same brief one-on- one assessment. Fall and spring assessments are reported to show progress over course of year. 3-day regional training sessions for trainers from 2,000 Head Start programs. They train local staff to conduct child assessments.

5. 5 How NRS Works (2) Child’s responses recorded by assessor on scannable answer sheets. Internet-based Computer-Based Reporting System (CBRS) supports assessment management and use. Programs may import data from local information-management systems into CBRS.

6. 6 NRS Data Processing Local Staff Enter Data into CBRS Xtria CBRS data received into Internet data base Local Assessors Complete Direct Child Assessments Pearson Answer Forms Scanned Westat HSNRS data received into analysis data base and analyzed Head Start Bureau receives and reviews HSNRS report Local Programs Regional Offices CBRS generates ID numbers based on class rosters

7. 7 HSNRS Assessment Components English Language Screener Vocabulary Letter Naming Early Math Skills

8. 8 Spanish Assessment Components Spanish Language Screener Spanish Vocabulary Spanish Letter Naming Spanish Early Math Skills

9. 9 Criteria for test components Skills that Congress and the President expect Head Start children to learn. Can be reliably measured in relatively brief assessment. Contribute to achievement in school. Can be enhanced by activities in Head Start. Skills that parents want children to learn.

10. 10 Developing Assessments Westat used extensive assessment data base from longitudinal studies on samples of Head Start children (FACES, QRC interventions, Head Start Impact Study) Select subsets of items with difficulty levels suitable for Head Start children. Simplify assessment procedures to enhance reliability with local assessors.

11. 11 Developing Assessments (2) Development process guided by expert advice of Technical Work Group. National field test, public comment preceded first year of implementation. Extensive quality-control effort, including monitoring study by independent contractor and comparisons with longitudinal sample survey (FACES 2003).

12. 12 Field Test National field test of 36 Head Start programs and 1,430 children conducted in April and May of 2003 Test of child assessment and “training of trainers” approach Parallel assessments by local Head Start staff and trained and experienced Westat field assessors

13. 13 Field Test Results All proposed components had acceptable internal consistency reliability. All components except phonological awareness (Elision task) had acceptable inter-assessor reliability. Good agreement between mean scores from experienced assessors & local staff. Similar means from teachers and non- teachers.

14. 14 Field Test Results (2) Language screener worked well with children who spoke a non-English language at home. Technical Work Group recommended deletion of Elision task, but approved moving forward with remainder of NRS battery. HSB decided on national implementation.

15. 15 National Implementation Summer “training of trainers” held in 7 cities and Puerto Rico. 2,831 local staff members trained and certified as English trainers. 576 staff certified as Bilingual trainers. Trainers returned to their programs and trained and certified more than 20,000 local assessors.

16. 16 First Year Data Collection Approximately 429,000 answer forms successfully processed in Fall 2003 and 410,000 in Spring 2004. Assessment data processed for 1,790 Head Start programs. Matching fall and spring assessments obtained for 343,260 children (80%). Largest assessment of preschool children ever carried out.

17. 17 How Assessments Are Scored Each assessment component scored separately. No attempt to produce single summary score for entire assessment. Mean percent of items correct and percentage of children at different skill levels reported to local programs. IRT scale scores developed for cross- program analysis and reporting to national and regional offices of Head Start Bureau.

18. 18 Growth Analysis Results Head Start children showed significant growth in all English-language skill areas. Reliability of program-level fall-spring growth rates based on NRS data was good for both English and Spanish assessments. Virtually no program showed zero growth or negative growth in English skills.

19. 19

20. 20

21. 21

22. 22

23. 23 Spanish Growth Analysis Head Start children assessed in Spanish showed growth in all skill areas. Growth was greater in programs in Puerto Rico, where instruction is in Spanish. Spanish growth rates comparable to English, except for vocabulary. Some programs on U.S. mainland showed average Spanish growth rates at or below zero.

24. 24

25. 25

26. 26 Current Activities Second year of national implementation of NRS underway. Thus far, numbers comparable to first year. Improvements in training and assessment procedures in response to feedback from Quality Assurance Study, TWG, and local staff. Distribution of materials proceeding in smoother, more timely fashion.

27. 27 Ongoing Analysis Continuing scrutiny of reliability and validity of NRS data. Multilevel regression analysis of spring achievement levels and fall-spring growth rates (program, center, class, child). Using independent variables from CBRS and Program Information Report. Qualitative study of high- and low- performing programs.

28. 28 Future Plans Possible expansion of NRS to encompass domains of phonological awareness, social-emotional development, child health. Exploration of alternative assessment techniques (e.g., computer-assisted personal assessment). Feasibility of using sampling to reduce assessment burden.

29. 29 In Conclusion Despite considerable controversy, a National Reporting System for Head Start has been implemented and appears to be working reasonably well. The system involves in-person assessment of all 4- and 5-year-old children on a limited but important set of early literacy and math skills.

30. 30 In Conclusion (2) Assessment results will be scrutinized carefully for several years before NRS data are used for any administrative decisions affecting individual programs. The Head Start Bureau will be examining the feasibility of expanding the scope of the NRS to encompass other domains, and use of alternative methods to improve accuracy and reduce burden.