The systematic examination and assessment of features of a program and its effects. Evaluation is designed to produce information that can be used by those who have an interest in the improvement or effectiveness of an initiative or program.
To assess the effects and value of a program ◦ for accountability ◦ for progress ◦ to identify best practice ◦ as a planning tool ◦ for research
the program target groups and needs realistic achievable objectives is the program ready for evaluation?
Identify needs State goals and measurable objectives Assess progress towards objectives
Efficacy Intervention tested under ideal conditions with optimal study design Effectivenes Field based intervention testing whether the intervention works as intended for a defined population in the real world Efficiency Results in comparison to the efforts required to obtain them – often cost or cost effectiveness
IMPACT OUTCOME Measures activities, program quality, reach Measures long term effects (goals) PROCESS Formative Measures short term objectives Develop program & resources
What is likely to work Best practices in health promotion Idea or concept development Developing and pilot testing materials Focus groups
Focus groups and interviews with partners about their needs Reviewing best practices from other programs such as Agita Sao Paulo Pilot testing in one company
Process evaluation ◦ Program implemented as intended ? ◦ Quality of program components ◦ Attendance rates ◦ Audience awareness of message ◦ Appropriate message learned ◦ Participant satisfaction
Companies enrolled Workshops held Promoters trained Companies participating in community events
Impact & Outcome Evaluation ◦ Short and long term effects of program ◦ Hierarchy of effects : what level of effects indicates program success ◦ Assessing the impact upon those who attend or participate ◦ Community effects
Physical activity level of employees PA level in community Disease rates or costs
Program Evaluation Framework Physical Activity Evaluation Handbook
Program Evaluation Framework Physical Activity Evaluation Handbook 1.Engage Stakeholders Include people with a vested interest in the program: implementers, partners, participants, possible opponents and decision makers. 2.Clearly Describe the Program The problem being addressed, descriptions of the planning, implementation and maintenance phases of the program, and finally, defining and communicating how all phases work together (eg. the "logic model").
Program Evaluation Framework Physical Activity Evaluation Handbook 3.Clearly Focus Asking a clear, straightforward question and ensuring linkage to the problem being addressed. Includes clarifying process and/or outcome evaluations of interest 4. Develop and Implement Data Collection Design of data collection efforts, selection of appropriate control groups, designation of appropriate baseline values and decisions about how much time should be built in to adequately measure "change”
Program Evaluation Framework Physical Activity Evaluation Handbook 5.Develop Conclusions Data analyses and interpretation. 6. Communicate Promotes the importance of tangible products from the evaluation, dissemination of those products, and follow-up (as appropriate).
All physical activity interventions should be evaluated. Evaluation begins with program planning There are many good programs, but very few are well evaluated Involve stakeholders Community – University partnerships can be very useful
Process as well as outcome evaluation Quantitative and qualitative Tailor evaluation to stakeholders and situation Evaluation is often undervalued compared with “hard science” Adequate resources should be devoted to evaluation. 10% is recommended
Evaluation results should be used to adjust programs and inform policy Share results with stakeholders Academic and practical training are needed The CDC Physical Activity Evaluation Handbook provides a good framework