1. Potential and Pitfalls of Experimental Impact Evaluation: Reflections on the design and implementation of an experimental Payments for Environmental Services project in Latin America Jos Vaessen, Maastricht University and University of Antwerp David Todd, GEF Evaluation Office Gert van Hecken, University of Antwerp

2. Outline 1.Introduction 2.Strengths and weaknesses of the experimental design and implications for analysis 3.Conclusions

3. Background Growing debate on the use of (quasi-)experimental methods in IE GEF EO decided to commission an evaluation of experimental design in practice as part of its annual report on impact RISEMP project: one of the few recently completed GEF projects with underlying experimental design 1. Introduction

4. RISEMP project Pilot project on PES in agricultural landscapes Regional project in three countries (2002- 2008) 8.72m US$ of which 4.77m US$ GEF funding Implemented by WB; three national NGOs 1. Introduction

5. (Research) objectives RISEMP project To demonstrate and measure: –effects of payment incentives for environmental services (PES) on adoption of integrated silvopastoral farming systems in degraded pasture lands –resulting improvements in ecosystems functioning, global environmental benefits, and local socio-economic gains 1. Introduction

6. Experimental design RISEMP project (in principle) random assignment of farmers to different ‘treatment’ groups Purpose was to test: –Effectiveness of payments: PES – CG –Effectiveness of technical assistance: PES + TA – PES only –Effectiveness of payment modality: PES 4yrs – PES 2yrs Annual monitoring of land use changes 1. Introduction

7. Objectives of the study To assess the strengths and weaknesses of the experimental design underlying the RISEMP project; its design and implementation; To assess the potential of the experimental design as a basis for analyzing the effectiveness of project incentives on land use changes; To suggest alternative and/or complementary methods for impact assessment; To draw lessons on the viability and utility of (quasi-) experimental designs as a future evaluation component in similar projects as the RISEMP project, in which GEF could play a role 1. Introduction

8. Conceptual framework for analysis: potential threats to (internal) validity Selection bias Treatment diffusion Unintended behavioral responses –Expected behavior or compliance behavior –Compensatory equalization –Compensatory rivalry Other aspects 1. Introduction

9. Methodology and delimitation Focus on Nicaraguan pilot site Interviews with stakeholders  Best source of information: field staff Interviews with farmers in pilot site Document review Challenge: how to get accurate information on design, implementation of experimental design and possible threats to validity?  semi-structured interviews, triangulation between sources 1. Introduction

10. Validity of difference in means: PES – CG Selection bias - Treatment diffusion Unintended behavioral responses - Conclusion: comparison of difference in means is invalid indicator of effect of payments in case of Nicaraguan pilot site 2. Strengths and weaknesses of the experimental design

11. Difference in means: PES - CG Figure 2. Incremental ESI points per hectare (2003-2007), three countries Source: own calculations based on RISEMP project data, January 2008 Note: PSA refers to pagos por servicios ambientales, or PES. 2. Strengths and weaknesses of the experimental design

12. Validity of difference in means: PES+TA - PES only Selection bias ~ Treatment diffusion - Unintended behavioral responses Conclusion: comparison of difference in means is invalid indicator of effect of technical assistance in case of Nicaraguan pilot site 2. Strengths and weaknesses of the experimental design

13. Validity of difference in means: PES2yrs – PES4yrs Selection bias Treatment diffusion Unintended behavioral responses Conclusion: comparison of difference in means is valid and useful indicator of effect of payment modality in case of Nicaraguan pilot site 2. Strengths and weaknesses of the experimental design

14. Difference in means: PES2yrs – PES4yrs Source: own calculations based on RISEMP project data, January 2008 Δ ESI pts 2. Strengths and weaknesses of the experimental design

15. How to make the experimental design work in practice Clarity and shared vision on what experiment is about Training and quality control of experiment Separate budget 3. Conclusions

16. Experimental method as an approach for assessing impact and complementary methods Does not fully ‘cover’ the scope of impacts as defined by the DAC Better suited to assess outcomes such as changes in land use and short-term ecological and socio-economic effects, less suitable for addressing: –long-term ecological and socio-economic effects and sustainability of effects –unintended effects –institutional and replicatory effects “The positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. These effects can be economic, sociocultural, institutional, environmental, technological or of other types” (OECD- DAC, 2002: 24). 3. Conclusions

17. …continued… Theory-based analysis of processes of change can complement the experimental analysis by helping to: –uncover causal mechanisms of change and variables affecting processes of change that are of interest –explain how policies (payments) affect types of subjects (farmers) in different ways 3. Conclusions

18. On the use of (quasi-)experimental designs in RISEMP and similar projects (Quasi-)experimental methods have strong comparative advantage in addressing the attribution problem in evaluation research Lessons for practice:  Criteria (on e.g. applicability and desirability) for selecting interventions for (quasi-) experimental analysis are needed (WHEN)  Appropriate budget and strategic plan for (quasi-) experimental design is needed (HOW)  Analysis should be complemented with other methods (HOW) 3. Conclusions

19. THANK YOU jos.vaessen@metajur.unimaas.nl jos.vaessen@ua.ac.be