1. Performance Measurement in the Conservation Community Status, progress, barriers, and next steps Background material for the Measuring Conservation Effectiveness Summit May 5-6th, Palo Alto, CA

2. INTRODUCTION To prepare for the May 5-6th summit on Measuring Conservation Effectiveness, 29 conservaiton implementers and funders were surveyed to ask about their Systematic Performance Measurement (SPM) practice, which we defined as: The regular monitoring, evaluation, and adaptation of conservation actions based on clearly stated goals, objectives, and assumptions so as to assess effectiveness, promote learning, and report achievements.

3. INTRODUCTION Respondents included: 15 conservation implementers, representing more than 2 billion US$ in conservation spending and an effort equivalent to 16,000 staff person years. 14 conservation funders, representing 1.1 billion US$ in annual support spread across about 1,000 grants. Respondents varied in size, but most had budgets between 11 and 100 M USD.

4. INTRODUCTION Respondents included: Conservation Implementers Conservation Funders Marisla Foundation Mulago Foundation

5. “OECD-DAC principles for evaluation” “project reviews against set criteria” “annual plans and performance reviews” “information flow up to organization level” “implementation of the Open Standards” INTRODUCTION In practice, implementers say SPM takes many forms… “peer-review audits” “reporting of key deliverables against expected results”

6. INTRODUCTION …as do funders. “constant and direct assessments of progress” “qualitative approach to evaluating grantee results and sustained change” “venture capital type investments with a clear theory of victory” “rigorous post-grant monitoring” “tool to understand why a funded project fails” “requirement for grantees” “communication tool for our board”

7. We note that small organizations seem to do SPM to a greater extent than large organizations and this affected our analysis. We show results herein both by % of responding organizations and % of total combined conservation spending, and graphics are coded accordingly. For example, we asked: At the scale of projects & programs, is SPM practiced well at your organization? INTRODUCTION By OrgBy $$$

8. 1. SPM is widely acknowledged as important... 2....but is not done to a great extent across our community… 3. …however, we can improve by leveraging the progress we have made… 4.and overcoming several critical obstacles. Here‘s what we learned INTRODUCTION

9. SPM IS IMPORTANT IN CONSERVATION 1. SPM is widely acknowledged as important in conservation.

10. Strongly disagree Strongly agree Moderately disagree Moderately agree Survey Question: In general for each audience, how strongly do you agree/disagree with the following statements: By Org SPM is generally viewed favorably and as a priority. SPM IS IMPORTANT IN CONSERVATION By Org

11. When SPM is implemented, the primary driver is to improve effectiveness. Survey Question: If your organization does or attempts SPM, how important are the following reasons behind that effort? By Org SPM IS IMPORTANT IN CONSERVATION

12. To assess and improve our effectiveness, there are many important questions we want to answer.... Survey Question to IMPLEMENTERS. How important is answering the following questions within your organization? By Org SPM IS IMPORTANT IN CONSERVATION

13. Survey Question to FUNDERS: How important is answering the following questions within your foundation? By Funders SPM IS IMPORTANT IN CONSERVATION

14. While funders are fairly confident they can answer key performance questions regarding their grantees… By Funders Survey Question: Using your foundation's current SPM system(s), how well can your foundation answer the following questions? SPM IS IMPORTANT IN CONSERVATION

15. …many of the implementing organizations feel less able to do so. Survey Question: Using your organization's current SPM system(s), how well can your organization answer the following questions? By Org By $ SPM IS IMPORTANT IN CONSERVATION

16. SPM IS NOT DONE WIDELY 2. That we often cannot answer key performance questions with confidence is likely because......SPM is not done to a great extent across our community.

17. For every conservation dollar spent, we say that only 10-30 cents worth is guided by SPM. SPM IS NOT DONE WIDELY Survey Question: Of the total budget for your organization’s conservation efforts, what % is guided by SPM?

18. SPM IS NOT DONE WIDELY Despite this estimate, we generally think we think we are doing SPM pretty well... By Org By $$$ By Funders

19. …but when we are pushed, we admit that we rarely put our projects through a full SPM cycle. only ~350-500 only ~2500 Of ~7000 projects have good conservation plans in place, but… currently undertaken by implementing organizations, have completed the cycle of design- implement-monitor-evaluate-adapt. SPM IS NOT DONE WIDELY

20. We generally do well at basic design and implementation of projects, but things break down when it comes to rigorous design, and monitoring, evaluation, and adaptation. SEE NEXT 2 SLIDES

21. DESIGN IMPLEMENT MONITOR EVALUATE & ADAPT SHARE By Org Survey Question: For your projects, indicate the extent to which your organization does each of the following SPM practices well.

22. By $$$ These patterns seem even more apparent in larger implementing organizations. DESIGN IMPLEMENT MONITOR EVALUATE & ADAPT SHARE

23. KEY INGREDIENTS TO PROGRESS TO DATE 3. To improve our ability to measure effectiveness, we can leverage several ingredients key to our progress to date.

24. Institutional mandate and SPM champions have been key ingredients to the progress we’ve made to date. By Org KEY INGREDIENTS TO PROGRESS TO DATE Survey Question: Where you do see SPM happening in your organization, how important were the following 'key ingredients' or 'catalysts' to SPM adoption?

25. CRITICAL OBSTACLES TO PROGRESS By Org Survey Question: Is SPM mandated at your organization? Is it practiced well at your organization? Institutional mandate is a key driver of SPM adoption.

26. The 4 implementers that say that 80-100% of spending is guided by SPM reinforce the importance of institutional mandate, evidence that SPM helps, and champions. Survey Question: Where you do see SPM happening in your organization, how important were the following 'key ingredients' or 'catalysts' to SPM adoption? By Top Implementers KEY INGREDIENTS TO PROGRESS TO DATE

27. Funders also emphasize the importance of a mandate, evidence that SPM helps, having a workplan that includes SPM, SPM being integral to an org’s mission and goals, and dedicated SPM funding. By Funders: Perception of Grantees

28. CRITICAL OBSTACLES TO PROGRESS 4. To improve our ability to measure effectiveness, we also must overcome several critical obstacles. For all organizations, these primarily include: Lack of time Lack of money Lack of staff dedicated to SPM Perception that SPM is too complex In addition to these, obstacles particularly important in larger organziations include: Lack of donor pressure Lack of board pressure Lack of demand from upper management

29. CRITICAL OBSTACLES TO PROGRESS Funding dedicated to SPM represents 1-5% of total conservation spending by implementers. By $$$ By Org

30. For every 100 projects, there is ~1 staff member dedicated to SPM at implementing organizations. Other staff data: Collectively, there’s about 1 conservation coach for every 60 projects and 150 organizational staff. There’s about 1 conservation trainer for every 140 projects. There’s about 1 auditor for every 500 projects. At organizations, there is a ratio of 230 : 1 organizational staff to SPM staff At foundations, the ratio is 30 : 1 There’s about 1 foundation staff dedicated to SPM for every 90 grants At foundations, about 20% of conservation program staff’s time is dedicated to SPM CRITICAL OBSTACLES TO PROGRESS 578 foundation staff By Funders 19 dedicated SPM staff 16,400 organizational staff By Org 71 dedicated SPM staff

31. Strongly disagree Strongly agree Moderately disagree Moderately disagree Survey Question: In general for each audience, how strongly do you agree/ disagree with the following statements? By Org $$$ CRITICAL OBSTACLES TO PROGRESS Understanding of what SPM is and how it might benefit an organization seems weak.

32. CRITICAL OBSTACLES TO PROGRESS The following four slides show responses to the survey question: In your experience, how important are the following factors in impediting adoption of SPM? The data is presented four ways: Slide 33: By Implementing Organization responses Slide 34: By Implementing Dollars spent Slide 35: By Implementing Organizations that report less than 20% of their budget is guided by SPM Slide 36: By Funder responses: perception of grantees

33. Money, time, and trainers are the most important barriers. SPM: Obstacles to SPM SPM Survey - Page By Org

34. SPM Survey - Page By $$$

35. By the 5 implementers that say <20% of budget is guided by SPM

36. By Funders: Perception of Grantees

37. 1. SPM is important in conservation  We say it is important and a top priority for the near future.  We believe answering key questions regarding our effectiveness is very important.  We generally do not believe we can answer these questions, however. 2. SPM is not done widely  Only 10-30% of current conservation spending is guided by SPM.  Very few projects do more than the initial steps of SPM  Basic design and implementation of projects is generally done well, but things break down when it comes to rigorous design, monitoring, evaluation, and adaptation. CONCLUSIONS Performance Measurement in the Conservation Community

38. 3. We can improve by leveraging key factors to progress to date... Performance Measurement in the Conservation Community 4....and by overcoming several critical obstacles. CONCLUSIONS For all organizations, these primarily include: Lack of time Lack of money Lack of staff dedicated to SPM Perception that SPM is too complex In addition to these, obstacles particularly important in larger organziations include: Lack of donor pressure Lack of board pressure Lack of demand from upper management  Institutional mandate  Champions for SPM within organizations  A vision for what can be accoplished with SPM  Evidence of SPM leading to increased effectiveness of conservation action

39. THANK YOU!! Frog illustrations © Ted Kahn 2010Ted Kahn Original artwork used with permission Neotropical Conservation Foundation www.neotropicalconservancy.org Thank you to all survey respondents! Summit Research Committee Sheila O'Connor (chair) Bernd Cordes William Crosse * Brett Jenks Richard Margoluis Matthew Muir *** Elizabeth O'Neill ** Nick Salafsky * Kristin Sherwood ** * = special effort Marisla Foundation Mulago Foundation