Module 5 - Questions and Criteria for Evaluations

27/10/20152 Questions and criteria What needs to be evaluated Methods and techniques Quality Aligning Questions and Methods

27/10/20153 Role of Questions in Evaluation Expression of a concern in its simplest form, a need felt by the sponsor (future user?) Asked in common language Best way for donors and evaluators to understand each other and agree Questions are refined using interpersonal communication tools (listening, rewording, etc.)

27/10/20154 Checking the Feasibility of Evaluation Questions Specific: is there a topic and a stem for each question? Measurable: can the question be answered? Actionable: will the answers provide the information needed for decision-making? Relevant: “nice to know” or “need to know”? Timely: is it important to ask this question now?

27/10/20155 What Can an Evaluation Tell Us? Success –Are we achieving the results that we were hoping for? –Are results being produced as planned at all levels of our logic model? Relevance –Do stakeholders care about what we are doing? Are we solving the problem? Cost-Effectiveness - Are we getting value for money?

27/10/20156 When Formulating Questions Ensure a clear link between questions and the issues of the evaluation Be realistic in the number of questions asked Choose questions selectively Ensure questions can be answered Focus on the important issues

27/10/20157 Sorting Questions Properly There are only 3 types of questions: … that require the use of different methods:

27/10/20158 Questions and Techniques Descriptive Available data Case studies Statistical survey Normative Available data Case studies Statistical survey Cost-benefit analysis Cost-effectiveness analysis Impact-focused Search for causal relations Statistical analysis Forecast analysis

27/10/20159 Evaluation Criteria Coherence Relevance Efficiency Effectiveness Impact Sustainability Replicability Goals, Purposes Main Objective Direct Results Activities Means Logical structure

27/10/ The Criteria What is their use ? The ‘politically correct’ side of the evaluation, based on the DAC’ standards In discussions/reflections on evaluation questions, criteria help ensure nothing is forgotten and helps sort out concerns A ‘total’ evaluation (on all criteria) reinforces the logical aspect of the findings, and therefore makes them sounder.

27/10/ Criteria and Questions: Correspondence Relevance Was the program well suited or adapted to reach the overall goal? Is the goal still relevant today? Does the program still meet the expectations of the stakeholders? Coherence In the program’s original design, were the objectives coherent (and are they still) with each other, did the planned activities make it possible to reach the operational objectives that were set, did the planned means make it possible to undertake the planned activities within the allocated time?

27/10/ Efficiency How did the project or the policy work? Were the resources properly mobilized? Were the unit costs within the standard range? Were the expected immediate results obtained and under what conditions in terms of quality, costs and time frame? Were the results satisfactory given the amounts of money invested? Were the contractual demands met? Criteria and Questions: Correspondence

27/10/ Criteria and Questions: Correspondence Effectiveness To what extent has the key objective of the project been reached? Has the program had the expected results from the point of view of the beneficiaries (addressing unbalances, reinforcing institutions, changing behaviours, etc.)? Impact Did the program help address the targeted problem (poverty level, for example) or produce the expected development? Did it have unforeseen negative impacts? Are its overall consequences positive?

27/10/ Criteria and Questions: Correspondence Sustainability Will the effects of the project or program be sustainable? Are the mechanisms put in place viable? Replicability Can the policy be duplicated ? What would it take?

27/10/ To ensure that all the questions have a concrete content, it is necessary to go back to the original objectives and indicators (project document and logical framework). When there is no logical framework, the evaluator must reconstruct it. Summary