1. Capacity Building For Program Evaluation In A Local Tobacco Control Program Eileen Eisen-Cohen, Maricopa County Tobacco Use Prevention Program Tips for Best Results For best results with evaluation planning, consider the following tips:  Invest heavily in planning.  Communication is key! Meetings may have to occur with project teams and individuals on an ongoing basis.  Lots of personal communication is needed with all levels of staff. Make sure line staff as well as management understand evaluation goals and the benefits to the organization.  Use humor.  Streamline data collection systems for staff ease whenever possible.  When training staff on evaluation issues it’s not always beneficial to review the whole forest, just one “park” is often enough.  Expand the evaluator’s role to be the intermediary between: other evaluators, programmers, and program staff.  Combine evaluation needs for data collection and what program managers need for program monitoring.  When there’s going to be a change…introduce it early and often in different settings.  Consider the objective of the evaluation at every step along the way.  Refer to your logic model to be certain that you have considered all parts of the program.  Plan to evaluate early and often. It is less costly to correct problems as they occur. The sooner we detect and correct problems, the more likely we are to have a successful program overall. 5 References: 1. Centers for Disease Control and Prevention. (1997). Smoking-attributable mortality and years of potential life lost—United States, 1984. MMWR 46 (20), 441–51. http://www.cdc.gov/mmwr/preview/mmwrhtml/00047690.htm 2. Centers for Disease Control and Prevention. (2001). Describe the program. In Introduction to program evaluation for comprehensive tobacco control programs (pp. 21-48). Atlanta, GA: the Author. http://www.cdc.gov/tobacco/evaluation_manual/ch2.html. 3. TTAC website http://www.ttac.org/power-of-proof/setting_stage/eval_defined.html. 4.Centers for Disease Control and Prevention. (2001). Introduction. In Introduction to program evaluation for comprehensive tobacco control programs (pp. 5-13). Atlanta, GA: the Author. http://www.cdc.gov/tobacco/evaluation_manual/introduction.html 5.Child Outcomes Research and Evaluation Team. (n.d.). Why evaluate your program? In The program manager's guide to evaluation. Retrieved January 25, 2004 from the Administration for Children and Families web site: http://www.acf.hhs.gov/programs/core/pubs_reports/chapter_1_pmguide.htm Evaluation Overview Tobacco use is the single most preventable cause of death and disease in the United States, contributing to more than 430,000 deaths annually. 1 Many tobacco prevention and control programs do tremendous work that is never fully recognized by the public, by other health professionals, or even by the people who benefit directly from the program’s accomplishments. Why does this happen? Usually, it is because program managers and staff strongly believe that their work is doing good things, but have no solid evidence to prove their success to people outside their program. In other words, such programs are missing one important element: evaluation. Program evaluation with local tobacco control programs is a necessary function towards achieving programmatic goals. In many programs, however, evaluation is an after-thought to program planning and implementation, particularly when evaluation is not the highest priority. Moreover, tobacco control staff may not have the requisite skills, resources, familiarity, and desire to conduct evaluation. Evaluation allows us to monitor program implementation compared to the program design. Additionally, evaluation encourages us to examine all the parts of a program, including what we put into the program, which activities take place, who conducts the activities, and who is reached as a result. 2 Evaluation provides us with information to serve a variety of purposes, including:  Learning whether proposed program materials are suitable for the people who are to receive them.  Learning whether program plans are feasible before they are put into effect.  Ensuring that a program is being conducted as it was designed.  Having an early warning system for problems that could become serious if unattended.  Monitoring whether programs are producing the desired results.  Learning whether programs have any unexpected benefits or problems.  Enabling managers to improve service.  Monitoring progress toward the program’s goals.  Producing data on which to base future programs.  Demonstrating the effectiveness of the program to the target population, to the public, to others who want to conduct similar programs, and to those who fund the program. 3 The CDC suggests that a good evaluator have these characteristics: Has experience in the type of evaluation needed. Is comfortable with qualitative and quantitative data sources and analysis. Is able to work with a wide variety of stakeholders. Can develop innovative approaches to evaluation while considering the realities affecting a program. Incorporates evaluation into all program activities. Understands both the potential benefits and risks of evaluation Educates program personnel about designing and conducting the evaluation. Will give staff the full findings (i.e., will not gloss over or fail to report certain findings for any reason). Has strong coordination and organization skills. Explains material clearly and patiently. Respects all levels of personnel. Communicates well with key personnel. Exhibits cultural competency. Delivers reports and protocols on time. 4 Department of Public Health Funded by Arizona Department of Health Services