1. Introduction to Comprehensive Evaluation
Justice Nonvignon, PhD
School of Public Health,
University of Ghana, Legon
July 2016
International Workshop on Impact Evaluation of Population, Health and Nutrition Programs

2. Acknowledgments
Sangeeta Tikyani Singh and Juan Pablo Gutierrez (INSP Mexico) contributed to this presentation.
Materials were adapted from MEASURE Evaluation and INSP Mexico materials.

3. Comprehensive: One Word, Multiple Potential Meanings Regarding Evaluation
Comprehensive in terms of scope (i.e., evaluating a sector, broad policy, government)
Comprehensive in terms of approach (i.e., multiple perspectives)
Comprehensive in terms of methods
Comprehensive in terms of the evaluatee (programme, policy, intervention, institution)

4. What kind of comprehensiveness is desirable?
What is relevant for the evaluatee?
Depends on where in the cycle of operation
How better address relevant questions?
Perspective, tools, methods that are most suitable for a particular question

5. What are policies meant for?
Ideally: Decision making to improve welfare
Identify a condition that requires attention
Design interventions/ programs / policies
Implement interventions/ programs / policies
What to do and how to do it?
Planning
Real life
It is important to contextualize this evaluation view. This has been developed thinking on public programes and policies, and under the general view that decision making in the public context is related to improving welfare. So, when a condition that is decided that requieres attention is identified, there is design process that then will be follow by the implementation. This is intendent to translate what was planned to the real life

6. Before effectiveness: design & operation
Identified causes
Best available evidence
Operation
Implementation according to design
Standardization
Results
Accomplishment of goals
Effectiveness
So, in our context, that most of the focus is put on final results, i.e. the program is reaching what was intended to or not, we wanted to highlight that before thoses results can or cannot be reached, it is very important to discuss design and operation

7. Why evaluate? Decide allocation Learning Improve performance
Evidence-based policy:
Improving efficiency
Improving effectiveness
Why evaluate?
Decide allocation
Provide evidence of destination and results of public funds
Accountability
Learning
I am sure that this is something that you have your own way to present it, but this has been useful for us in thinking the different questions an evaluator may answer. We evaluate, of course, because we want to learn, as we are an academic institution. But the demand for evaluation is driven by the need for useful information for the programs. Policies and programs are evaluated because there is a need to generate evidence to inform the decision making process. We promote evaluation because we are also promoting evidence based policy and because we are convinced of the relevance of evaluation for the accountability process.
Identify best practices and opportunity areas
Test alternatives within existent programs
Transferability to other contexts
Modifications to existent programs.
Assessment of continuation.
Improve performance

8. Comprehensive evaluation

9. A general framework to connect different types of evaluations
Basics of comprehensive evaluation
Evaluation phases mirroring programs life-cycle
Linking program phases from evaluation perspective
It is proposing evaluation as a stepwise building
The design of a program is its foundations, identify what is intended to achieve
Define what indicators should be used to measure success
So, our approach is to look at the different evaluation needs that are related with the different phases of a program, and linking thoses phases from an evaluation perspectie. That is, seen it as a continuos.

10. A framework built on existing research
Comprehensive evaluation as a process
Each type of evaluation is an input for the next phase of evaluation (and of the program)
i.e. evidence from an evaluation phase is needed for the next phase

11. A definition for comprehensive evaluation
A dynamic, interactive and progressive process that aims to analyze program performance using synergically a set of analytical tools that allow to identify areas of improvement considering formulation, planning and operation of programs, and providing recommendations oriented to improve programs
It includes diagnosis, design, processes, implementation, results, effectiveness, efficiency

12. Validation of identified problem and it´s causal chain
Comprehensive evaluation framework
Características de la evaluación integral
Validation of identified problem and it´s causal chain
Analysis of diagnosis

13. Comprehensive evaluation framework
Características de la evaluación integral
Consistency between design and causal chain, target population definition
Design evaluation
Analysis of diagnosis

14. Comprehensive evaluation framework
Características de la evaluación integral
Verification of resources allocation
Evaluation of targeting and coverage
Design evaluation
Analysis of diagnosis

15. Comprehensive evaluation framework
Analysis of resources and information flows for program initiation
Características de la evaluación integral
Evaluation of implementation
Evaluation of targeting and coverage
Design evaluation
Analysis of diagnosis

16. Comprehensive evaluation framework
Validation of operation according to planning
Evaluation of processes
Características de la evaluación integral
Evaluation of implementation
Evaluation of targeting and coverage
Design evaluation
Analysis of diagnosis

17. Comprehensive evaluation framework
Goals and performance analysis
Evaluation of results
Evaluation of processes
Características de la evaluación integral
Evaluation of implementation
Evaluation of targeting and coverage
Design evaluation
Analysis of diagnosis

18. Comprehensive evaluation framework
Measurement of attributable effect
Evaluation of effectiveness
Evaluation of results
Evaluation of processes
Características de la evaluación integral
Evaluation of implementation
Evaluation of targeting and coverage
Design evaluation
Analysis of diagnosis

19. Comprehensive evaluation framework
Measurement of efficiency (allocation, technical, administrative)
Comprehensive evaluation framework
Evaluation of efficiency
Evaluation of effectiveness
Evaluation of results
Evaluation of processes
Características de la evaluación integral
Evaluation of implementation
Evaluation of targeting and coverage
Design evaluation
Analysis of diagnosis

20. Comprehensive evaluation framework
Evaluation of efficiency
Evaluation of effectiveness
Evaluation of results
Evaluation of processes
Características de la evaluación integral
Evaluation of implementation
Evaluation of targeting and coverage
Design evaluation
Analysis of diagnosis

21. Measuring and accountability
Evidence based design
Implementation according to planning
Measuring results
THIS IS THE PURPOSE OF COMPREHENSIVE EVALUATION

22. How to state evaluation questions?
Understanding the evaluatee
What it is for? What is the theory behind? What is it trying to accomplish?
Understanding evaluatee needs
In what phase is the evaluatee? What evidence will help the evaluatee to improve its performance?

23. This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. Views expressed are not necessarily those of USAID or the United States government.