1. American Evaluation Association Annual Conference
Value of the Rapid Evaluation Method Leah Robin, Marian Huhman, and Catherine Rasberry
American Evaluation Association Annual Conference
Orlando, Florida
11/11/2009
The findings and conclusions in this report are those of the authors and do not necessarily represent the
official position of the Centers for Disease Control and Prevention.

2. What are rapid evaluations? Examples of findings and impact
Overview
What are rapid evaluations?
Examples of findings and impact
Lessons learned

3. Purpose of rapid evaluations
To evaluate innovative programs that are unlikely to be evaluated

4. Characteristics of rapid evaluations
Conducted with established, innovative programs, initiatives, or policies
Chosen from nominated programs
Provided as TA to sites and involves questions of interest to them
Involve non-experimental or quasi-experimental methods
Data collection lasts months

5. Well-established program Focus on school-aged youth
Characteristics of programs, initiatives, and policies that undergo rapid evaluation
Well-established program
Focus on school-aged youth
Likely to make a difference
Ability to have comparison sites/groups
Address under-served populations and health disparities

6. Products from rapid evaluations
Reports to the selected programs
Presentations prepared for the selected programs
Presentations and briefings to:
Federal agencies
State and local partners
National conferences and meetings
Peer-reviewed publications
Success stories

7. Rapid evaluation steps
Nominations of projects
Selection of projects for evaluability assessment
Completion of evaluability assessments
Selection of project for rapid evaluation
Conduct evaluation
Design
Recruitment
Data collection
Data analysis
6. Dissemination of findings

8. Rapid evaluation teams
CDC
Collaboration across CDC
Contractors
ETR Associates
ICF Macro
USDA and NIH
Local agencies and universities

9. Support provided to programs for rapid evaluations
Incentives for participation
District-level
School-level
Participants
Establishing an evaluation coordinator position
Hired to work with the program
Usually chosen by program staff
Assists in recruiting, data collection, and data management
Hiring local data collection staff

10. Local agency responsibilities
Participate in a month evaluation
Approve all aspects of evaluation study
Assist with recruitment of study participants
Assist in recruiting local site coordinator
Facilitate data collection
Approve report and manuscript

11. Select evaluation sites
CDC responsibilities
Select evaluation sites
Facilitate activities between contractor and school district
Provide evaluation technical support
Participate in determining evaluation design
Prepare reports and manuscripts for dissemination

12. Evaluation contractor responsibilities
Design evaluation in collaboration with CDC and local agency
Help sites select and contract with local site coordinator
Conduct evaluation over months
Site recruitment
Data collection
Data analysis
Prepare reports and manuscripts for dissemination

13. Evaluations of interventions
Michigan: Exemplary Physical Education Curriculum
Mississippi: Fresh Fruit and Vegetable Distribution Pilot
Albuquerque, NM: Asthma
management program
Charlotte-Mecklenburg, NC: Asthma management program

14. Sequential PE curriculum
Example of an evaluation of an intervention: Exemplary Physical Education Program (MI)
Sequential PE curriculum
K-12
Based on motor skill development
Follows NASPE and Michigan content standards
Quasi-experimental design
2 year data collection with baseline and 3 follow-ups
16 Michigan elementary schools statewide
4th and 5th grade students (N = 1464)

15. Exemplary Physical Education Program (MI) findings and impact
Positive results relative to comparison group
Confidence in motor skills (4th grade)
Physical activity knowledge (5th grade)
Level of motor skills (4th and 5th grade)
Minutes of physical activity (4th grade)
Energy exerted during physical activity (4th grade)
Impact
Program improvements
Teacher training improvements
Evidence of utility of motor skills training focus

16. Formative Evaluations
Rhode Island: Healthy Vending Initiative
Pueblo, Colorado: Pricing Initiative
Seattle, WA: Safe Schools Project

17. Example of a formative evaluation: Rhode Island Healthy Vending Initiative
Multiple component initiatives across all districts
Develop comprehensive school wellness policies
Offer healthier foods in a la carte and vending machines
Healthy breakfast meetings target district superintendents
Interviews with 72 district administrators in 31 districts
Case studies in 6 school districts (5 high schools and 6 middle schools) representing high and low implementers of initiative
Data collection included observations, interviews, questionnaires, and focus groups

18. Rhode Island Healthy Vending Initiative findings
Schools with approved nutrition policies in place had more healthy snack items available for sale a la carte and in vending machines than those without such policies
An abundance of less healthy food options were still available to students from multiple sources throughout the day
A positive and dynamic partnership among the schools, the district, and the food service providers enhanced understanding of the best approach to putting written policies into practice
School level involvement of students, parents, and teachers, early in the policy development process increased awareness and acceptance of policies

19. Rhode Island Healthy Vending Initiative impact
Better creation and implementation of local wellness policies across Rhode Island
Expanded the healthy vending initiative to include multiple stakeholders at the school and district level
Provide TA and written materials to USDA for developing and evaluating local wellness policies

20. Lessons learned Widespread public nomination process helpful
Many programs are not ready for evaluation
Evaluability assessments are critical
Several need to be conducted
Must include discussions with multiple stakeholders
Outside perspectives helpful
RWJF process
Summaries of nominations and follow-up by phone
Expert panels to determine who receives evaluability assessment

21. Lessons learned Local participation is critical
Timelines
Workload
Incentives
Local expertise is needed in
Identifying local issues
Developing measurement instruments
Recruitment of districts and schools

22. Broad impact Fostered collaborations across CDC and other agencies
Impacted evaluation on a national level
Impacted policies and practices on a state level
Provided information to the field on program improvement and new directions for programs, initiatives, and policies
Improved selected programs, initiatives, and policies
Web page developed with rapid evaluation information, success stories, and measures

23. American Evaluation Association Annual Meeting
Value of the Rapid Evaluation Method Leah Robin, Marian Huhman, and Catherine Rasberry
American Evaluation Association Annual Meeting
Orlando, Florida
11/11/2009
The findings and conclusions in this report are those of the authors and do not necessarily represent the
official position of the Centers for Disease Control and Prevention.