ICT4D: Evaluation Dr David Hollow DSV, University of Stockholm

Objective To understand the importance of effective monitoring, evaluation and impact assessment within ICT4D programmes

● What it is and why it matters ● Evaluation trends ● Challenges and opportunities ● Principles for effective work ● Where to find out more

Definitions

Monitoring: A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an on- going development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds (OECD).

Evaluation: The systematic and objective assessment of an on- going or completed project, program or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision- making process of both recipients and donors (OECD).

Impacts: Positive and negative, primary and secondary, long-term effects produced by a development intervention, directly or indirectly, intended or unintended (OECD).

Why does this matter?

Nothing matters more!

‘The fact that somebody in the room would say the impact is unclear is to me amazing - unbelievably amazing. There's not a person in this room who hasn't bought a laptop for their child, if they could afford it … So there's only one question on the table and that's how to afford it? That's the only question. There is no other question - it's just the economics.’ (Negroponte, 2009)

or…

‘The positive impact of ICT use in education has not been proven … despite thousands of impact studies, the impact of ICT use on student achievement remains difficult to measure and open to much reasonable debate.’ (Trucano, 2005)

Types of evaluation and evaluation tools Baseline testing Indicators Control groups Focus groups Interviews Observations Pre-test Post-test Quantitative Qualitative Mixed-methods Formative Summative

Trends ● Millennium Development Goals (and post- MDG) ● ‘Value for Money’, ROI, RBM ● Demand for evidence-based programming

Monetary, quantitative, qualitative… ‘We need to know if what you are proposing is good: if you can monetise it, monetise it, if you can’t monetise but can count it, then count it, if you can’t monetise or count it but can observe it, then observe it. If you can do all three then do all three.’ (DFID interview 2012)

Evaluation is useful in ICT4D because it… Evaluation is harmful in ICT4D because it … Makes people accountable Distracts from real purpose of programme Highlights successes Imposes simplistic notions of success Facilitates learning Gives no time to implement learning Enables effective allocation of resources Allocates resources only where impact can be quantified Gives stakeholders a voicePressurises stakeholders

Key issues: ● What is being evaluated and why? ● What are the appropriate indicators (input, output, outcome)? ● What are the appropriate methods? ● Are baselines and control groups required? ● What are the likely challenges / ethical concerns?

Theory of Change

Evaluation principles ● Embrace complexity - systems evaluation ● Learn from the beginning – process based evaluation ● Learn from everyone – participatory evaluation ● Understand environment – contextual evaluation Technology is our tool – transformation is our objective – evaluation helps us to discern what is good

Evaluation principles ‘Not everything that can be counted counts, and not everything that counts can be counted.’ (Attributed to Einstein)

Evaluation communities ● Monitoring and Evaluation News ● The Pelican Initiative: Platform for Evidence-based Learning and Communication for Social Change ● The Outcome Mapping Learning Community ● Reflect and Learn ● DAC Network on Development Evaluation ● AEA365