1. Jindra Cekan, PhD Valuing Voices, Sustainable Solutions for Excellent Impact Wageningen University M&E for Responsible Innovation Conference, March 2015 Who’s Listening? Community-led Post- Project Sustainability Evaluation

2. Great Evaluation and Learning work © 2015 Valuing Voices, Cekan Consulting LLC Our industry is evaluates thousands of multi-sectoral projects each year Donors have invested millions of dollars/Euros in learning ventures looking for real impact, such as: Grand (Health) Challenges Making All Voices Count Aid Transparency (IATI) Impact Evaluation by 3ie 100 national national evaluator associations in 93 countries support country-led evaluation

3. Overlap of M&E for Responsible Innovation + Valuing Voices How can M&E responsibly support the management and governance of innovation … and contribute to deeper reflexivity and transparent decision making? Are Institutional changes needed? How do we take responsibility for systemic change in: – M&E professional’s roles & responsibilities; – M&E process design, focus and approach Three questions: 1.What is in our way? 2.What can be done? 3.What methods are best? © 2015 Valuing Voices, Cekan Consulting LLC

4. Governance: Global Aid Effectiveness © 2015 Valuing Voices, Cekan Consulting LLC Paris Declaration (2005), Accra Agenda for Action (2008) and the Busan Partnership (2011) created shared principles to achieve common goals: Ownership of development priorities by developing counties A focus on results Partnerships for development Transparency and shared responsibility

5. Development becoming more transparent, focused on sustainability Local Accountability has been proposed as a core feature of the new post-2015 development agenda, according to UNICEF. post-2015 development agenda Donors talk about Country-led Development, M&E Capacity (OECD), Local Systems (USAID), as do non-profits like Local First- wonderful. Let’s support community-owned and driven ones!Country-led Development, M&E Capacity Local Systems Local First © 2015 Valuing Voices, Cekan Consulting LLC MDGs are becoming Sustainable Development Goals post-2015 Sustainable Development Goals

6. Changes in Responsibilities, Approach, Focus © 2015 Valuing Voices, Cekan Consulting LLC For them to evaluate our efforts = how must we change?

7. 1. What’s in our way? © 2015 Valuing Voices, Cekan Consulting LLC

8. Definitions: ‘Sustainable’ Development? © 2015 Valuing Voices, Cekan Consulting LLC How do we know it’s sustainable if we do not go back and ask? o $1.6 trillion spent on international development by EU and US alone since 2000 o $100 billion spent in 2014 o Trillions more spent by private donors ALL UNEVALUATED o National evaluations nonexistent o Data + tracking systems discarded <1% of projects evaluated post- closeout for sustainability We don’t know, we haven’t asked about the other 99% projects

9. Scant Design for self-sustainability Communities’ voices are rarely elicited to design what activities they will make last… for RFPs are not designed in communities Yet to not evaluate sustainability for future design short-changes the communities which have entrusted us to improve their lives © 2015 Valuing Voices, Cekan Consulting LLC

10. What Exists Now in project sustainability 2-5 year duration, typically with great M&E …The continuity of people’s lives in communities… ? ? ? ? ?Earlier or concurrent Development projects ? ? ? ? ? STOP ? baselin e midterm final E V A L U A T I O N S Feedback loops?! © 2015 Valuing Voices, Cekan Consulting LLC Follow –on or new Project GAP

11. “Responsible Development” requires Innovation © 2015 Valuing Voices, Cekan Consulting LLC

12. Why We Value Voices © 2015 Valuing Voices, Cekan Consulting LLC Designing for sustained exit should be our first priority We ALL deserve self- sustaining projects: Communities/ countries, we development workers, taxpayers… it’s true respect. ‘They’ should evaluate our assistance for effectiveness, not the other way around A business model works – looking at Return on Investment through their eyes illuminates what should be repeated – or not. Once we know what communities could self- sustain, let’s design and fund future projects accordingly! ‘They’ are ‘Us’

13. Views from outside on our DIME* © 2015 Valuing Voices, Cekan Consulting LLC "You cannot set your own examination, take the examination and mark it. Then you cry out success or failure. The community will just 'look at you' and wonder what is the issue.... It will be sustainable development if the people at community level are involved in designing and delivering their own dreams of development” Peter Kimeu, Catholic Relief Services’ Senior Technical Advisor for Partnership/ Solidarity, East Africa (30 years expertise) * Design, Implementation, Monitoring and Evaluations

14. Views from outside on our DIME* © 2015 Valuing Voices, Cekan Consulting LLC “We push new ideas, rather than building on what communities have already created…. we must be able to look back and see what they've learned and can sustain themselves.… to show what they've done themselves, and where they think they're taking themselves. [Sustainability] is an ignored area. The challenge is, it's just been gathering numbers for 5 years, 10 years…. We don't have the right feedback system from communities to the NGO world. It’s the tyranny of deadlines so they "just get their concerns" when staff go out [to the field]… ” R.K. Kenyan Independent Evaluation Professional (20 yrs) * Design, Implementation, M&E

15. VV Data Sustainability Approach: Open Data Design data captured using open data international standards (such as IATI) – Aggregate with other data sets (project, demographic, maps) – Sharable beyond project staff/activities – Available after project period of performance – Allows for public accountability /transparency – Historical marker of “what happened back then” for learning from measurement © 2015 Valuing Voices, Cekan Consulting LLC Mapping malnutrition in Uganda

16. 2. What Can Be Done? © 2015 Valuing Voices, Cekan Consulting LLC

17. Discussion: what is in our way, what can be done? © 2015 Valuing Voices, Cekan Consulting LLC World Bank Civil Society Forum Meeting 2014 1.ARE WE RIGHT ABOUT WHAT’S IN OUR WAY? 2.WHAT CAN WE DO TO HELP DONORS LISTEN AND HELP IMPLEMENTERS COME ON BOARD? LISTENING TO YOUR VOICES!

18. 3. What methods are best? © 2015 Valuing Voices, Cekan Consulting LLC

19. Right Design of Post-Project Sustainability Evaluation and Objectives © 2015 Valuing Voices, Cekan Consulting LLC DEVELOPMENT INDUSTRY LEARNING ACCOUNTABILITY TO PARTICIPANTS LEARN FROM UNEXPECTED IMAPCTS EMPOWER LOCAL NGOS, COMMUNITIES, AND NATIONAL EVALUATORS LOCAL ROI SHAPES WISER FUNDING DECISIONS AND BETTER PROGRAMMING PROMOTE COMMUNITY-INFORMED AND JOINTLY LED LEARNING FROM SUSTAINED OUTCOMES

20. Moving from Forest to Weeds… How © 2015 Valuing Voices, Cekan Consulting LLC “Because… the devil is in the detail”

21. Site/ Partner Selection Project sites closed within the last 2-5 years Communities, partners available for consultations, final evaluations accessible National evaluators are available and interested Sampled sites comparable in livelihood, diversity No similar work done by NGOs in gap years Ideally web-savvy partners to handle data/ analysis/ curation/ storage Analysis, presentation, sharing and curation of findings are clear (by whom and for whom, how, where, for how long) © 2015 Valuing Voices, Cekan Consulting LLC

22. Mixed-Methodologies © 2015 Valuing Voices, Cekan Consulting LLC Qualitative: Participatory RRA methods/ tools (based on Appreciative Inquiry and Empowerment Evaluation, potentially also Outcome Harvesting) of past participants in projects Quantitative (purposive, then random-sampled HH survey, possibly including mobile surveys for participants who’ve left Small control group: non-participants in same communities Use mix of internal and external evaluation teams covering all sectors Findings shared widely and stakeholders invited to discuss

23. Qualitative process © 2015 Valuing Voices, Cekan Consulting LLC Research: 1.Update sectoral evaluator team and translators in RRA approach 2.Discuss best RRA tools (maps/calendars, rankings, focus groups) 3.Visits: Two to three days per site in 3-4 sites, with meetings in communities reminding them of project and asking: A.What activities/outputs were most self-sustainable and why? B.Which activities should be repeated? Which shouldn’t? C.What else would participants like to evaluate of the project? Distillation: 1.Daily team analysis 2.Each site summary analysis is presented to the community for input 3.Community invited to final presentation in region 4.Findings inform survey + report Later, compare across projects/sectors

24. Quantitative process © 2015 Valuing Voices, Cekan Consulting LLC Research: 1.Survey questionnaires of a random sample subset of the population (past participants and some communities’ non- participants) 2.Stratification of intervention area by criteria; the strata would include a homogeneous stratum (same behaviors in project, e.g. all sustaining activity) and heterogeneous stratum (those not) but who are judged on the same key variables or indicators. 3.Randomly choose a fixed number of villages for interviews Distillation: 1.SPSS for analysis 2.Community invited to final presentation in region 3.Findings for integrated report Later, compare across projects/sectors

25. Looking for Unexpected Results © 2015 Valuing Voices, Cekan Consulting LLC

26. LWR Niger post-project unexpected impacts Evaluated Food security/ Livelihood/ Resilience project: sheep distribution, fodder, wells, peaceful transhumance for 500 women post-drought Expected findings: Variable impacts on women’s economic benefits Great impact on time gain from wells for income generation. Unexpected wonder: Decreased domestic violence from presence of livelihood + water © 2015 Valuing Voices, Cekan Consulting LLC

27. 3. Methods: Discussion questions © 2015 Valuing Voices, Cekan Consulting LLC 1.COMMENTS ON METHODS FOR ANY ONE SITE? 1. THOUGHTS ON COMPARABILITY ACROSS SITES, ACTIVITIES AND PROJECTS?

28. Let’s change how international development works… together © 2015 Valuing Voices, Cekan Consulting LLC Thank you for your input! Jindra@ValuingVoices.com