1. Outcome-based Planning and Evaluation Gloria Latimer, Ed.S, Director of Community Programs Jason Vahling, M.P.H., Community Program Specialist

2. *MMWR, Vol. 48, No. RR-11 Recommended Framework for Evaluation in Public Health Practice*

3. Logic Model Engages evaluation stakeholders Educates stakeholders about realistic expectations for change Describes your program Focuses your evaluation design

4. Logic Model (continued) Directs your data gathering efforts Justifies your conclusions Communicates evaluation results

5. Logic Models Input: A resource dedicated to or used by the program. Activity: Type of service the program provides to fulfill its mission. What the program does with the inputs-how it goes about transforming them into products. Output: The direct product of program operation. Outcome: Benefit to participants during or after participating in the program.

6. Logic Model INPUTSACTIVITIES OUTPUTS OUTCOMES RESOURCES money staff volunteer equipment CONSTRAINTS laws regulations funders’ requirements SERVICES medicine assistance training education counseling mentoring transportation services case management PRODUCTS classes taught counseling sessions conducted educational materials distributed hours of service delivered participants served meds distributed BENEFITS FOR PEOPLE new knowledge increased skills changed attitudes or values modified behavior improved condition altered status Short -Term Intermediate Long -Term

7. Identifying Outcomes Envision the expected impacts of the program on its participants/audience Ask yourself – “What has changed for individuals, organizations, or the community as a result of this program?” Look beyond what the staff does and ask “So What?” Use the “So That” chain of logic

8. “So That” Chain of Logic Teacher training & development of school-based curriculum Increased awareness, knowledge, attitudes, and skills related to youth initiation So that Decreased social norms that support tobacco use among youth So that Reduced initiation among youth So that

9. Selecting Outcomes to Measure Which outcomes are most important & most closely related to your core business? Which outcomes are the most meaningful to the participants? Which outcomes will provide the best information for decision-making & program improvement?

10. Selecting Outcomes to Measure (continued) Which outcomes can the program be expected to influence in a non-trivial way? Which outcomes are most likely achievable given the resources available?

11. Colorado’s Logic Model

12. Community Public Health Programs are Essential Components of STEPP’s Program Local public health agencies have core functions For social norms to change, involvement must extend to where people live, work and play Policy is an effective public health strategy and voluntary and legislated policy interventions are supported (clean indoor air, tobacco free schools, illegal sales to minors as examples) Industry influence can be more effectively blocked at local level

13. The Challenge Community Programs: Funding to 46 (from 8 in 2000) public health agencies serving 63 counties in the state through non-competitive grants Coordination: – Federal/State/State Partners/Local Partners Collaboration – Defining roles – Avoiding duplication

14. National State Local Colorado’s Experience: Linking Outcomes Across Points of Intervention

15. Statewide Objectives Based on Colorado’s Logic Models Provides Colorado with the ability to broadly evaluate program effectiveness Demonstrates accountability to program stakeholders including state and local officials, policymakers, and community leaders Policy focused

16. Statewide Objectives…cont. Ensures that programs adhere to the recommended CDC framework Maximizes training and technical assistance resources by focusing on several key goals & objectives Facilitates additional regional and county level data collection

17. National: Increase to 100% the proportion of worksites with formal policies that prohibit smoking or limit it to separately ventilated areas. State: Increase the proportion of worksites in Colorado that have an official policy that prohibits smoking: In public areas by 20% to 85%. In work areas by 20% to 98%. Local: By June 30, 2003, increase by/to X% the proportion of worksites that adopt formal policies that prohibit smoking. Example: Secondhand Smoke Linked Objective

18. National: Increase to 75% the proportion of adult smokers who stopped smoking for a day or longer in the last year. State: Increase the proportion of smokers who report that they: Quit smoking for 1 day or longer in the past 12 months by 50% to 66%. Quit smoking cigarettes regularly 6-12 months ago by 25% to 5.6%. Local: By June 30, 2003, increase by/to X% the proportion of quit attempts among adult tobacco users in X city/county. Example: Cessation Linked Objective

19. National: Increase to 51% the proportion of jurisdictions with a 5% or less illegal buy rate among minors. State: Increase by 10% the number of retailers who refuse to sell tobacco products to minors. Local: By June 30, 2003, by/to X% the proportion of tobacco retailers that sell tobacco products to minors in X city/county. Example: Youth Linked Objective

20. Conclusion Logic models helped guide Colorado’s programmatic and evaluation decisions Logic models were used to develop Colorado’s statewide objectives Statewide objective help to link outcomes at the national, state, and local levels

21. Assumptions Every local program had the capacity to evaluate the outcomes proposed by STEPP The framework allowed local programs enough flexibility to feel vested in the decision making process because they could choose their own activities Local programs had familiarity with the logic model A database format would simplify the process

22. Lessons Learned Not all local programs have data collection capacity…strike a balance Involve local stakeholders in the decision making process It takes time for local partners to understand and apply logic model principles Not all local programs are technologically oriented

23. Questions, Comments, and Concerns jason.vahling@state.co.us 303-692-2578 THANK YOU! gloria.latimer@state.co.us 303-692-2513