1. Impact Evaluation: Initiatives, Activities, & Coalitions Stephen Horan, PhD Community Health Solutions, Inc. September 12, 2004

2. 2 Objectives Participants will learn: Participants will learn: –1. When an evaluation should be conducted –2. What questions to ask to address outcomes & impact –3. Strategies for evaluating outcomes & impact –4. Strategies for incorporating multiple programs and activities –5. Strategies for incorporating community input

3. 3 Ground Rules Evaluation talk can be confusing – don’t let semantics get in the way Evaluation talk can be confusing – don’t let semantics get in the way –This presentation defines  Outcomes as changes in health program clients  Impacts as changes in the organization, system, or community  Different funders or textbooks might define these terms differently –This presentation uses a model of personal change based on knowledge, attitudes & beliefs, skills, behaviors, and health status.  There are other equally valid models – no one is saying you have to use this one!

4. 4 When to evaluate 1. When a funder requires it 1. When a funder requires it 2. When the program is experimental 2. When the program is experimental 3. When the program is controversial 3. When the program is controversial 4. When the program is expensive 4. When the program is expensive 5. When you think the program might not be worth it 5. When you think the program might not be worth it 6. When you think the program is worth it but might be in danger of losing funding 6. When you think the program is worth it but might be in danger of losing funding

5. 5 Questions for addressing outcomes & impact

6. 6 Five phases of evaluation Five phases of evaluation –1. Context evaluation –2. Program design evaluation –3. Program implementation or process evaluation –4. Program outcome evaluation –5. Program impact evaluation

7. 7 Questions for addressing outcomes & impact 1. Context evaluation 1. Context evaluation –Why is this program needed? –Does this program duplicate other programs? –Does this program conflict with other programs? –How might public policy affect this program? –Does this program fit our funder’s criteria?

8. 8 Questions for addressing outcomes & impact 2. Program design evaluation 2. Program design evaluation –What is our program theory? –Who is our target population? –What are the needs of our target population? –What outcomes will we help out target population achieve? –What services will we provide? –How many people will we serve? –What resources will we need to run the program?

9. 9 Questions for addressing outcomes & impact 3. Program implementation evaluation 3. Program implementation evaluation –How many people are we serving? –What services are they receiving? –Are they receiving the services they should? –What activities are we conducting? –How much are we spending? –All of the above compared to Program Design

10. 10 Questions for addressing outcomes & impact 4. Program outcome evaluation 4. Program outcome evaluation –How have our clients changed in terms of:  Knowledge  Attitudes & beliefs  Skills  Behaviors  Health status

11. 11 Questions for addressing outcomes & impact 5. Program impact evaluation 5. Program impact evaluation –How has our organization or community changed in terms of:  Financing  Service capacity  Organization of services  Quality of services  Accessibility of services  Cost of services  Community health status

12. 12 Strategies for evaluating outcomes & impact

13. 13 Strategies for evaluating outcomes & impact 1. Begin with your program theory 1. Begin with your program theory 2. Develop your program design 2. Develop your program design 3. Capture your program design in a logic model 3. Capture your program design in a logic model 4. Decide which outcomes & impacts to measure 4. Decide which outcomes & impacts to measure 5. Negotiate outcomes & impacts with your funder 5. Negotiate outcomes & impacts with your funder

14. 14 Strategies for evaluating outcomes & impact 1. Begin with your program theory 1. Begin with your program theory –Program theory is your set of assumptions about cause and effect between our program and are target population’s outcomes –Most often overlooked component of program evaluation –What makes you think this will work?

15. 15 Strategies for evaluating outcomes & impact 2. Capture your program design in a program logic model 2. Capture your program design in a program logic model –Target population –Assumptions (program theory) –Inputs –Activities –Outputs –Outcomes –Impact

16. 16 Strategies for evaluating outcomes & impact 3. Develop your program design 3. Develop your program design –Specify your target population –Specify your program theory in a set of assumptions –Specify your target impact –Specify your target outcomes –Specify your target outputs –Specify your required activities –Specify your required resources

17. 17 Strategies for evaluating outcomes & impact 4. Decide which outcomes & impacts to measure 4. Decide which outcomes & impacts to measure –Is this outcome or impact important to our audiences? –Is this outcome or impact within our sphere of influence? –Is this outcome or impact achievable within the timeframe of the reporting period? –Is this outcome or impact measurable?

18. 18 Strategies for evaluating outcomes & impact 5. Negotiate with your funder 5. Negotiate with your funder –Not every outcome or impact can be measured –Not every outcome or impact should be measured –Arm yourself with knowledge  Your program theory  Why a particular outcome or impact should or should not be measured  What it will cost to measure a particular outcome or impact –If they insist, tell them what you think the evaluation will cost, and ask them to either fund it or suggest a cheaper way to do the evaluation

19. 19 Strategies for incorporating multiple programs and activities

20. 20 Strategies for incorporating multiple programs and activities 1. Get everyone to agree on: 1. Get everyone to agree on: –Program theory –Target outcomes –Target impacts 2. Get everyone to develop a program design complete with logic model 2. Get everyone to develop a program design complete with logic model 3. Agree upon common outcome indicators 3. Agree upon common outcome indicators 4. Agree upon common impact indicators 4. Agree upon common impact indicators 5. Share data collection tools 5. Share data collection tools 6. Share data analysis methods 6. Share data analysis methods 7. Pool the data or agree on a standard format for aggregating the data 7. Pool the data or agree on a standard format for aggregating the data

21. 21 Strategies for incorporating community input

22. 22 Strategies for incorporating community input 1. Ask community members to review and comment on your program logic model 1. Ask community members to review and comment on your program logic model 2. Ask community members to review and comment on your evaluation workplan 2. Ask community members to review and comment on your evaluation workplan 3. Ask community members to pilot test your data collection instruments 3. Ask community members to pilot test your data collection instruments 4. Survey your community on its perceptions of your program 4. Survey your community on its perceptions of your program 5. Ask community members to help you interpret your evaluation results 5. Ask community members to help you interpret your evaluation results

23. 23 Additional resources

24. 24 Additional resources For additional ideas, guidelines, and connections, go to www.CommunityHealthResource.net For additional ideas, guidelines, and connections, go to www.CommunityHealthResource.net www.CommunityHealthResource.net In particular: In particular: –Go to the Program Design page and Program Evaluation page and download worksheets

25. 25 Presenter Stephen Horan, PhD Stephen Horan, PhD President President Community Health Solutions, Inc. Community Health Solutions, Inc. 9603 BC Gayton Road Suite 201 9603 BC Gayton Road Suite 201 Richmond, VA 23238 Richmond, VA 23238 T: 804.673.0166 T: 804.673.0166 F: 804.673.2244 F: 804.673.2244 shoran@communityhealthinfo.com shoran@communityhealthinfo.com shoran@communityhealthinfo.com