1. Performance Measurement in the Conservation Community Status, progress, barriers, and next steps Elizabeth O‘Neill, Conservation Auditor, WWF International & Matt Muir on behalf of the Measuring Conservation Effectiveness Summit Research Committee

2. INTRODUCTION Purpose: Survey of 29 conservation implementers and funders to ask about their Systematic Performance Measurement (SPM) practice. Implementors: “OECD-DAC principles for evaluation” “annual plans and performance reviews” “peer-review audits” “implementation of the Open Standards” Funders: “communication tool for our board” “rigorous post-grant monitoring” “constant and direct assessments of progress” “qualitative approach to evaluate grantee results” Plan AdaptDo Check

3. INTRODUCTION Respondents: 15 conservation implementers: ~2.1 billion US$ ~7,000 projects 16,000 staff 14 conservation funders: ~1.1 billion US$ ~1,000 grants 578 staff

4. 1. SPM is widely acknowledged as important... 2....but is not done to a great extent across our community… 3.…however, we can improve by leveraging the progress we have made… 4.and overcoming several critical obstacles. What we learned: INTRODUCTION

5. SPM IS IMPORTANT IN CONSERVATION 1. SPM is widely acknowledged as important in conservation.

6. SPM is generally viewed favorably and as a priority by most managers and boards. SPM IS IMPORTANT IN CONSERVATION Strongly disagree Strongly agree Moderately disagree Moderately agree ? ?

7. SPM IS IMPORTANT IN CONSERVATION When SPM is implemented, we say the primary driver is to improve effectiveness. Do we understand why projects fail? Are actions cost-effective? Are we having intended impacts? Are we adapting and improving? What can be learned to improve? Can we demonstrate credible results? 85 to 90% of all respondents say answering these questions is important

8. FUNDERS Do we understand why projects fail? Are actions cost-effective? Are we having intended impacts? Are we adapting and improving? What can be learned to improve? Can we demonstrate credible results? % of respondents SPM IS IMPORTANT IN CONSERVATION Not at all Very well Minimally well Somewhat well 0 100

9. FUNDERS ORGS Do we understand why projects fail? Are actions cost-effective? Are we having intended impacts? Are we adapting and improving? What can be learned to improve? Can we demonstrate credible results? SPM IS IMPORTANT IN CONSERVATION Not at all Very well Minimally well Somewhat well % of respondents 0 100

10. FUNDERS ORGS $$$ Do we understand why projects fail? Are actions cost-effective? Are we having intended impacts? Are we adapting and improving? What can be learned to improve? Can we demonstrate credible results? SPM IS IMPORTANT IN CONSERVATION Not at all Very well Minimally well Somewhat well % of spending 0 100

11. SPM IS NOT DONE WIDELY 2. That we often cannot answer key performance questions with confidence is likely because......SPM is not done to a great extent across our community.

12. For every conservation dollar spent, we say that only 10-30 cents worth is guided by SPM… SPM IS NOT DONE WIDELY Survey Question: Of the total budget for your organization’s conservation efforts, what % is guided by SPM?

13. …and only about 5% of projects go through a full SPM cycle (i.e., plan-do-check-adapt). only ~350-500 only ~2500 Of ~7000 projects have good conservation plans in place, but… currently undertaken by implementing organizations, have completed the SPM cycle SPM IS NOT DONE WIDELY

14. By $$$ Practice tends to break down at certain places in the cycle DESIGN IMPLEMENT MONITOR EVALUATE & ADAPT SHARE 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20      

15. KEY INGREDIENTS TO PROGRESS TO DATE 3. To improve our ability to measure effectiveness, we can leverage several ingredients key to our progress to date.

16. SPM: The importance of having a mandate Key ingredients to the progress we’ve made to date: KEY INGREDIENTS TO PROGRESS TO DATE (1)Institutional mandate (2)Presence of an SPM champion (3)A vision for what could be accomplished with SPM (4)Evidence that SPM helps (5)Dedicated SPM staff (6)Dedicated SPM funding

17. CRITICAL OBSTACLES TO PROGRESS 4. To improve our ability to measure effectiveness, we also must overcome several critical obstacles.

18. CRITICAL OBSTACLES TO PROGRESS OBSTACLE 1: LACK OF TIME TO DO SPM 90% of respondents indicate this is an extreme or major obstacle

19. CRITICAL OBSTACLES TO PROGRESS OBSTACLE 2: LACK OF MONEY Most implementers indicate they invest no more than 1-5% of total conservation spending in SPM. By Org % of annual conservation spending

20. OBSTACLE 3: LACK OF STAFF DEDICATED TO SPM CRITICAL OBSTACLES TO PROGRESS 578 foundation staff By Funders 19 dedicated SPM staff (30:1) By Org 16,400 organizational staff 71 dedicated SPM staff (230:1)

21. CRITICAL OBSTACLES TO PROGRESS OBSTACLE 4: FEELING THAT SPM IS TOO COMPLEX Strongly disagree Strongly agree Moderately disagree Moderately agree

22. OBSTACLE 5: LACK OF PRESSURE/DEMAND FROM ‘ABOVE‘ By Org By $$$ CRITICAL OBSTACLES TO PROGRESS Smaller organizations seem to do SPM to a greater extent than larger ones. Is SPM practiced well at your organization? Key obstacles identified: Lack of donor pressure Lack of board pressure Lack of demand from upper management

23. Insufficient time + Weak funding + Few staff + Limited understanding + Lack of demand from ‘above’ = An apparent low priority placed on SPM Unanswered research questions: Why is SPM a low priority? What are root causes? What will it take to make major advances? CRITICAL OBSTACLES TO PROGRESS

24. 1. SPM is important in conservation  We say it is important and a top priority.  We want to be able to answer key questions regarding effectiveness.  But we generally cannot answer these questions. 2. SPM is not done widely  Only 10-30% of current conservation spending is guided by SPM.  Very few projects do more than basic design plus implementation.  Things break down at more rigorous design and monitoring, evaluation, adaptation, and sharing learning. CONCLUSIONS Performance Measurement in the Conservation Community

25. 3. We can improve by leveraging key factors to progress to date... Performance Measurement in the Conservation Community 4....and by overcoming several critical obstacles. CONCLUSIONS  Time  Money  Dedicated SPM staff  Knowledge/understanding  Demand from ‘above’  Institutional mandate  Champions for SPM  A vision for what SPM can do  Evidence that SPM helps

26. THANK YOU!! Frog illustrations © Ted Kahn 2010Ted Kahn Original artwork used with permission Neotropical Conservation Foundation www.neotropicalconservancy.org Thank you to all survey respondents! Summit Research Committee Sheila O'Connor (chair) Bernd Cordes William Crosse Brett Jenks Richard Margoluis Matthew Muir Elizabeth O'Neill Nick Salafsky Kristin Sherwood