American Evaluation Association Annual Conference Value of the Rapid Evaluation Method Leah Robin, Marian Huhman, and Catherine Rasberry American Evaluation Association Annual Conference Orlando, Florida 11/11/2009 The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

What are rapid evaluations? Examples of findings and impact Overview What are rapid evaluations? Examples of findings and impact Lessons learned

Purpose of rapid evaluations To evaluate innovative programs that are unlikely to be evaluated

Characteristics of rapid evaluations Conducted with established, innovative programs, initiatives, or policies Chosen from nominated programs Provided as TA to sites and involves questions of interest to them Involve non-experimental or quasi-experimental methods Data collection lasts 12 - 24 months

Well-established program Focus on school-aged youth Characteristics of programs, initiatives, and policies that undergo rapid evaluation Well-established program Focus on school-aged youth Likely to make a difference Ability to have comparison sites/groups Address under-served populations and health disparities

Products from rapid evaluations Reports to the selected programs Presentations prepared for the selected programs Presentations and briefings to: Federal agencies State and local partners National conferences and meetings Peer-reviewed publications Success stories

Rapid evaluation steps Nominations of projects Selection of projects for evaluability assessment Completion of evaluability assessments Selection of project for rapid evaluation Conduct evaluation Design Recruitment Data collection Data analysis 6. Dissemination of findings

Rapid evaluation teams CDC Collaboration across CDC Contractors ETR Associates ICF Macro USDA and NIH Local agencies and universities

Support provided to programs for rapid evaluations Incentives for participation District-level School-level Participants Establishing an evaluation coordinator position Hired to work with the program Usually chosen by program staff Assists in recruiting, data collection, and data management Hiring local data collection staff

Local agency responsibilities Participate in a 12-24 month evaluation Approve all aspects of evaluation study Assist with recruitment of study participants Assist in recruiting local site coordinator Facilitate data collection Approve report and manuscript

Select evaluation sites CDC responsibilities Select evaluation sites Facilitate activities between contractor and school district Provide evaluation technical support Participate in determining evaluation design Prepare reports and manuscripts for dissemination

Evaluation contractor responsibilities Design evaluation in collaboration with CDC and local agency Help sites select and contract with local site coordinator Conduct evaluation over 12-24 months Site recruitment Data collection Data analysis Prepare reports and manuscripts for dissemination

Evaluations of interventions Michigan: Exemplary Physical Education Curriculum Mississippi: Fresh Fruit and Vegetable Distribution Pilot Albuquerque, NM: Asthma management program Charlotte-Mecklenburg, NC: Asthma management program

Sequential PE curriculum Example of an evaluation of an intervention: Exemplary Physical Education Program (MI) Sequential PE curriculum K-12 Based on motor skill development Follows NASPE and Michigan content standards Quasi-experimental design 2 year data collection with baseline and 3 follow-ups 16 Michigan elementary schools statewide 4th and 5th grade students (N = 1464)

Exemplary Physical Education Program (MI) findings and impact Positive results relative to comparison group Confidence in motor skills (4th grade) Physical activity knowledge (5th grade) Level of motor skills (4th and 5th grade) Minutes of physical activity (4th grade) Energy exerted during physical activity (4th grade) Impact Program improvements Teacher training improvements Evidence of utility of motor skills training focus

Formative Evaluations Rhode Island: Healthy Vending Initiative Pueblo, Colorado: Pricing Initiative Seattle, WA: Safe Schools Project

Example of a formative evaluation: Rhode Island Healthy Vending Initiative Multiple component initiatives across all districts Develop comprehensive school wellness policies Offer healthier foods in a la carte and vending machines Healthy breakfast meetings target district superintendents Interviews with 72 district administrators in 31 districts Case studies in 6 school districts (5 high schools and 6 middle schools) representing high and low implementers of initiative Data collection included observations, interviews, questionnaires, and focus groups

Rhode Island Healthy Vending Initiative findings Schools with approved nutrition policies in place had more healthy snack items available for sale a la carte and in vending machines than those without such policies An abundance of less healthy food options were still available to students from multiple sources throughout the day A positive and dynamic partnership among the schools, the district, and the food service providers enhanced understanding of the best approach to putting written policies into practice School level involvement of students, parents, and teachers, early in the policy development process increased awareness and acceptance of policies

Rhode Island Healthy Vending Initiative impact Better creation and implementation of local wellness policies across Rhode Island Expanded the healthy vending initiative to include multiple stakeholders at the school and district level Provide TA and written materials to USDA for developing and evaluating local wellness policies

Lessons learned Widespread public nomination process helpful Many programs are not ready for evaluation Evaluability assessments are critical Several need to be conducted Must include discussions with multiple stakeholders Outside perspectives helpful RWJF process Summaries of nominations and follow-up by phone Expert panels to determine who receives evaluability assessment

Lessons learned Local participation is critical Timelines Workload Incentives Local expertise is needed in Identifying local issues Developing measurement instruments Recruitment of districts and schools

Broad impact Fostered collaborations across CDC and other agencies Impacted evaluation on a national level Impacted policies and practices on a state level Provided information to the field on program improvement and new directions for programs, initiatives, and policies Improved selected programs, initiatives, and policies Web page developed with rapid evaluation information, success stories, and measures http://www.cdc.gov/healthyyouth/evaluation/rapid/index.htm

American Evaluation Association Annual Meeting Value of the Rapid Evaluation Method Leah Robin, Marian Huhman, and Catherine Rasberry American Evaluation Association Annual Meeting Orlando, Florida 11/11/2009 The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.