1. New frontiers Evaluation methods Theory of change Project cycle and risk management Jesper Johnsøn, CMI, U4 Bergen, February 4, 2014

3. Frontiers… Current plateau – (cross) country-level, bribery, fraud Evaluations of specific reforms Integration with risk management Proxy indicators

4. Imbalances and the “missing middle” Imbalances: – So far, resources have been invested in standardised diagnostic and advocacy tools, not learning and evaluation Diagnostics: follow trends – indicators, data, monitoring Evaluation: what works – question, design and methods, indicators, data, judgement Missing middle Output OutcomeImpact Individual Sector Country Level of results Indicator targets

5. When and how to evaluate corruption costs/anti-corruption effects? Diagnose corruption risk (does corruption matter, are patterns changing?) If yes, prepare for evaluation (how and how much does corruption matter, how effective is anti-corruption?) 1.Logic model: Think through how the initiative is intended to create results by contributing to behavioral, organisational, political, or societal change, expressed in a result chain or theory of change. 2.Monitor and evaluate: Work on increasing the evaluability of the reform by shaping design, establishing indicators, systematic data collection, baselines, comparison groups INDICATORS EVALUATION QUESTION, DESIGN, METHODS PROBLEM/ DIAGNOSIS DATA ANALYSIS, COMPARISON, JUDGEMENT, CHANGE

6. Overview of types of evaluations Impact evaluationsProgramme evaluationsProcess evaluations Focus: cause-and-effect, attribution, impact Methods: counterfactual, comparison groups, statistical analysis, randomisation Can answer question: What would have happened in the absence of the programme? Focus: effectiveness, contribution, outcomes Methods: comparative analysis, cost- effectiveness/benefit analysis, benchmarking Can answer question: Has the programme been effective in achieving its stated objectives? Focus: efficiency, implementation, internal mechanisms, outputs Methods: workflow analysis, compare performed vs planned activities and outputs. Can answer question: Is implementation done according to plan, and do results look attainable? Clear objectives Indicators Mixed methods Triangulation Controls Randomisation Comparisons Baselines Trends

7. Code of Conduct example, level of results

8. Code of Conduct example, building an impact story

9. Diagnostics, evaluation, risk management and the project cycle Framework for holistic use of different tools Shows interlinkages between diagnostics, risk management and evaluation tools

10. Diagnostics, evaluation, risk management and the project cycle http://prezi.com/osddirkyzzpw/?utm_campaign=sha re&utm_medium=copy&rc=ex0share http://prezi.com/osddirkyzzpw/?utm_campaign=sha re&utm_medium=copy&rc=ex0share

11. The Proxy Challenge

12. Challenges we confront Linked, but not inseparable: – Measuring changes in corruption levels – Evaluating whether anti-corruption efforts are successful Progress on measuring bribery and financial fraud, but two overall indicator problems remaining: – How to measure other types of corruption (patronage, conflict of interest, abuse of power, etc.) – How to present a measure of overall corruption levels in a country, region, sector, or organisation not biased towards measurable types of corruption, and can show trajectories of change

13. Why proxies? Good, existing “direct” indicators for bribery and financial fraud (need better application of methods) Can capture “missing middle” – provide needed innovation for some types of corruption, to capture trajectory of performance, and measure behavioural change “Proxy indicators are alternatives to direct indicators that more directly measure the phenomenon under study but that may be hard to operationalize or require overly costly data collection”

14. Criteria Standard criteria for indicator development (SMART) Reflect de facto changes and behavioral changes Prioritize sensitivity to context over standardization Measure specific types of corruption Ensure that changes in corruption trends are attributable to reforms

15. “To assemble a body of promising ideas…” Clearly define the proxy indicator (or indicators) you suggest as a good measure of anti-corruption results Identify the type of corruption measured by the proxy indicator Clarify how the proxy indicator reflects changes in corrupt behaviours Present ideas for how the validity of the proxy indicator can be tested Describe any relevant geographic, sectoral, or institutional context Explain how the indicator can be combined with other proxy and nonproxy (direct) indicators to obtain a better measurement of overall anti-corruption progress Comment on the strengths and weaknesses of the proxy indicator and the feasibility of its use by aid agencies, governments, civil society, and others for the purposes of monitoring and reporting