1. Welcome to Program Evaluation Overview of Evaluation Concepts Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. This work is licensed under a Creative Commons Attribution-NonCommercial- ShareAlikeLicense.

2. Introduction Objectives of course, What is Program Evaluation Types of evaluation (general) Example

3. Objectives of course By the end of the course, you will be able to: (1) Explain major concepts in program evaluation: – Type of evaluation and their purpose – Levels of measurement – Sources of data – Study designs, threats to validity

4. Objectives of course (2) Perform skills required in program evaluation: – Design a conceptual framework – Develop objectives, indicators – Conduct a focus group – Pretest a survey – Process statistics – Use participatory evaluation techniques (3) Write an evaluation plan

5. What’s not covered? How to do multivariate analysis required to analyze the designs discussed in class (e.g., randomized trial) Calculations of sample size, power Surveillance techniques Additional qualitative methods Actual implementation - ECUR 810

6. Tell me more Jay.

7. Overview of Program Evaluation

8. What is program evaluation? “the systematic assessment of the operation and/or outcomes of a program or policy, compared to a set of explicit or implicit standards as a means of contributing to the improvement of the program or policy.” (Wiess, 2009)

9. Which really means… Telling some group how well the program is doing what it is supposed to be doing. “There is room for judgment.” Brian Noonan

10. Why do we do program evaluation? To determine the effectiveness of a program: – Did it achieve its objectives? – Were effects similar across subgroups? To identify ways of improving on existing program design. Continue or terminate a program. To satisfy financial considerations, requirements. For ‘political’ reasons, PR, staff morale, communication.

11. Initially you need to think about and ask: What is the problem? Why am I doing this? What exactly do I want to know? Does an answer already exist? How do I find out? Who is involved? What will this cost? What will I do with the data? What happens when the evaluation is finished?

12. Tailoring evaluation to a specific context What questions will the evaluation answer? What methods/procedures will be used? What will be the evaluator-stakeholder relationship? – Independent evaluation – Participatory/collaborative evaluation – Empowerment evaluation What do you do with the answers?

13. Other key factors: program structure and circumstances Stage of program development Political context (conflict over goals) Structure of the program – Scope of activities, type of services – Number and location of service sites – Characteristics of intended audience Resources available: – Human, $$$, support of admin or management

14. Challenges in program evaluation Dealing with the ‘politics’ of programs Having the program design change mid-course Balancing the tensions between scientific soundness and practicality (utility for decision- makers) Obtaining $$$ and support for strong designs

15. Primary types of evaluation Formative – Needs assessment/diagnostic – Pretesting Process – Implemented well? As planned? – Input analysis – Output, effectiveness, user satisfaction Summative – Monitoring of program utilization – Monitoring of behavior or health status – Impact/outcome assessment Cost effectiveness, cost-benefit Benchmarks, standards

17. Characteristics of PE (Powell, 2006) Used for decision making. Deals with research questions about a program. Takes place in the real world of the program. Usually represents a compromise between pure and applied research. Should be carefully planned. Have a purpose not an end in itself. Must be hope for action or change. Can be used to raise internal as well as external awareness

18. Pallet Cleanser

19. An example… Two ways a programs can be studied. Two ways to determine what is working or not working with a program.

20. Implementation failure vs. Theory failure Implementation failure – Program is not implemented as planned Theory failure – Program is implemented as planned – Intervention does not produce intermediate results and/or – They do not produce desired outcome

21. Measures of outcome (Rossi, Lipsey, Freeman, 2004) Initial Intermediate Long-term

22. Example: mass media program to reduce HIV transmission Initial – Antecedents to behavior – Ex: knowledge, attitudes, practice Intermediate – Behaviors – Ex: contraceptive or condom use Long-term – Health status – Ex: fertility rates, HIV prevalence

23. Example: conceptual model of program impact

24. Implementation failure

25. Theory failure

27. What????