After-School Program Quality How Do We Know It When We See It? Theresa Ferrari, Lisa Lauxman, Ina Linville, Deirdre Thompson, Nancy Valentine CYFAR 2004 Pre-Conference May 11, 2004

Adapted from the Extension School-Age Child Care Consortium, 1993 Revised April 2003 Distributed at the 4-H Afterschool National Roll-Out Conference, St. Louis, MO

Welcome! Our Purpose Extension Cares Initiative System Update: “The Big Picture” Dr. Nancy Valentine

Extension CARES…for America’s Children and Youth The Big Picture

Overall Goal To increase the quality, availability, accessibility, affordability, and sustainability of child care, school-age care (after-school), and teen out-of- school programs How are We Doing?

Did You Know? Extension has provided 2,530 hours of training for 22,700 childcare, after-school, and teen out- of-school time staff, families, and community leaders valued at $202,400.*+ Over three-fourths found the training to be of high quality and would recommend it to others. *Based on 2,530 training hours reported x 3 hrs. preparation time for each hour taught x average salary calculation of $20.00 per hour. +Based on data from 18 states – 2003

Did You Know? Over three-fourths of those trained gained knowledge or skills, found the information to be useful, and would use the information. Almost 1,400 providers were trained and 7,000 children and youth were impacted by peer- reviewed Extension curricula. (Eight states reporting)

Did You Know? An additional 1,200 children and youth were served through the establishment of 90 new programs. (Two states reporting) About $5 million was leveraged by Extension to support programs. (Nine states reporting)

Did You Know? Almost 5,000 new children and youth were served in 54 new after-school programs that were established and managed by Extension. (Seven states reporting) More than 60 4-H clubs were started in after- school programs that served an additional 5,400 children and youth. (Eight states reporting)

ECI Levels of Evaluation Level 1 (single state or county) –Recording training sessions Level 2 (single state or county) –Client satisfaction evaluations –Semi-annual report of outputs, activities, and outcomes Level 3 Level 4 Level 5 Multi-state or county efforts

What Is the ECI Evaluation and Reporting System? A web-based system for reporting information from early care and education, school-age care (after-school), and teen out-of-school programs Data entry is easy and fast, and the system performs all the analysis for you. The system provides instant professional reports of your results.

Level 1- Recording Trainings A variety of information is collected on the session: –title, date, type of session –type of credit offered –training location, training hours –number of participants, intended target audience –age groups served by participants in the session

Level 2- Client Satisfaction Evaluations There are 4 versions of evaluations: –Provider/Staff –Families –Extension –Community You can enter evaluation data on the web and/or scanning services are available.

Semi-Annual Report Currently there are 12 questions on which to report program activities, outputs, and outcomes. You may select any or all of the questions on which to report. You are responsible for reporting data for each question. A semi-annual report can be generated twice a year from reported data.

What Are People Saying? Federal agencies and organizations are very interested in the information. Gives us critical information to articulate the breadth and depth of our work. Helps us position Extension for potential resources. Will give states the data needed for the next Plan of Work templates. Cathann Kress, Director, Youth Development, National 4-H Headquarters, USDA

What Are People Saying? “I love the ECI reporting and evaluation system. I use the data as a tool to improve the quality of the training I deliver.” Jennifer Miller, County Extension Educator, Ashe County, North Carolina

What Are People Saying? Flexible reporting and evaluative system that captures and records staff training activities Web-based and is fairly easy to master. Harry Mangle, Extension Educator, University of Connecticut Cooperative Extension

What Are People Saying? Data to report results to the J.C. Penney Afterschool Fund, John Deere, and other sponsors of 4-H Afterschool Critical for securing resources and positioning 4-H in the after-school market Don Floyd, President and CEO, National 4-H Council

What Are People Saying? Immediate & efficient “up to the minute” aggregated county data State-wide system already in place but easy to merge and use Using data for a State 4-H Department Review to show growth and quality Program development and improvement and to secure funding Excited to be part of a national effort and to see how NC compares to other states Rosa Andrews, Extension Associate and Coordinator, North Carolina 4-H School-Age Care Programs

What Are People Saying? At State Level: –Snapshot of programs and audiences reached by county educators –Report program impacts, identify program needs, looks at strengths, needs, and audience diversity –Explore other questions to ask –Very useful with other partners, policymakers, and funders At County Level: –Track programs, audiences, and effectiveness –Assist with annual performance appraisals Debbie Richardson, Child Development Asst. Specialist, Oklahoma Cooperative Extension Service, Oklahoma State University

Level 3 & 4 - Program Quality Goal: To assess program quality before measuring child/youth outcomes National Afterschool Association (a.k.a. NSACA) Standards & Accreditation System School Age Care Environmental Rating Scale (SACERS)

Level 5 – Research Design W.T. Grant Foundation proposal submitted State and Multi-State Research Projects Individual Research

Other Connections ES-237 National Outcomes & Indicators by FCS Next POW Cycle

Other Alignments CSREES Strategic Plan CSREES Office of Planning and Accountability REEIS NAE4HA

Scoring (1) Inadequate A rating of 1 is the maximum score given: –If any of the indicators under 1 is scored yes.

Scoring A rating of 2 is given: –If no part of 1 is scored yes and –If half or more of the indicators under 3 are scored yes.

Scoring (3) Minimal and (5) Good A rating of 3 or 5 is given: –Only if all of the indicators are met. All indicators in 3 must be met before any higher rating may be given for an item.

Scoring A rating of 4 or 6 is given: –When all the lower ratings are scored yes and –When half or more of the next higher indicators are scored yes.