Outcome Harvesting A tool for identifying and understanding results in complex circumstances Ricardo Wilson-Grau

Agenda Outcome Harvesting’s origins When is Outcome Harvesting a useful tool How does the tool work — exercise Six steps Methodological challenges

Inspired by the Outcome Mapping methodology. A tool for practitioners operating in dynamic, uncertain situations to monitor and evaluate the social change results they are achieving. e Harvesting Outcome Harvesting

Developed since 2003 by me and my colleagues Barbara Klugman, Claudia Fontes, Fe Briones Garcia, Gabriela Sánchez, Goele Scheers, Heather Britt, Jennifer Vincent, Julie Lafreniere, Juliette Majot, Marcie Mersky, Martha Nuñez, Mary Jane Real, Natalia Ortiz and Wolfgang Richert.

1. International social change networks

2. International development donors

What did all of these organisations have in common?

Planning Monitoring “I can honestly say that not a day goes by when we don’t use those evaluations in one way or another.” Evaluation

Planning Monitoring “I can honestly say that not a day goes by when we don’t use those evaluations in one way or another.” Evaluation

Why?

Relationships of cause and effect are KNOWN Simple 12

Annual polio vaccination campaign ACTIVITIES OUTCOMES IMPACT INPUTS Inspired by Jeff Conklin, cognexus.org Time OUTPUTS M&E of a simple intervention

ACTIVITIES OUTCOMES IMPACT INPUTS Time M&E of a simple intervention OUTPUTS ¿Are we effective? Are we efficient? Annual polio vaccination campaign

Relationships of cause and effect are UNKNOWN Complex

OUTPUT OUTCOME INPUTS ACTIVITY INPUTS ACTIVITY INPUTS ACTIVITY INPUTS OUTPUT ACTIVITY OUTPUT OUTCOME M&E of a complex intervention Time Strengthening nonviolent responses to communal conflict in the Horn of Africa

OUTPUT OUTCOME INPUTS ACTIVITY INPUTS ACTIVITY INPUTS ACTIVITY INPUTS OUTPUT ACTIVITY OUTPUT OUTCOME Time 17 Dimensions in which the relationships of cause and effect are known Not either or SIMPLE

OUTPUT OUTCOME INPUTS ACTIVITY INPUTS ACTIVITY INPUTS ACTIVITY INPUTS OUTPUT ACTIVITY OUTPUT OUTCOME Time COMPLEX Results are substantially unforeseeable. Not either or Unknown relations of cause and effect dominate.

Outcome Harvesting is NOT useful

Outcome Harvesting IS useful

When do you harvest outcomes? Time SummativeDevelopmentalFormative Progress of intervention

Track record Over 300 networks and associations, NGOs, community-based organisations, research institutes, and government agencies on all seven continents. Identifying and reporting almost 3,000 changes they have influenced.

Most recently In August, the UNDP identified Outcome Harvesting as one of eleven “innovative approaches to monitor and evaluate the performance of public policies, programmes and service delivery.”

SOCIAL ACTOR Individual Organisation Institution Group or community What is an outcome? 1. Social actors changing the way they do things Behaviour Relationships CHANGES Policies Practices

Example Change: In 2009, the Palestinian Authority revitalizes an employment fund for qualified people living in Palestine.

Cause Effect When is it your outcome? 2. A change that you influenced

Example of influence Change: In 2009, the Palestinian Authority revitalizes an employment fund for qualified people living in Palestine. Contribution: The Global Call to Action against Poverty (GCAP) coalition in Palestine launched the “Stand Up and Be Counted” campaign, which mobilized 1.2 million people in Working with the Ministry of Labor, the coalition also helped secure multilateral funding and technical assistance on the management of the fund.

First… Identify outcomes Paradigm shift

Second… Discover the activities and outputs that contributed Paradigm shift

Other information? Significance of the outcome Collaboration with other social actors Contribution of other actors and factors History Context Evidence of impact on people’s lives And so forth.

Example Change: In 2009, the Palestinian Authority revitalizes an employment fund for qualified people living in Palestine. Significance: Demonstration of how mass citizen action can be combined with the engagement of political decision makers to lead to transformative changes in government policy and practice.

Six steps of Outcome Harvesting Six steps: 1.Design the harvest 2.Review documentation and draft outcomes 3.Engage with informants 4.Substantiate 5.Analyse, interpret 6.Support use of findings

1. Design the harvest, focusing on usefulness Primary intended users Principal intended uses Useful Outcome Harvesting questions Credible data

Credible data = credible sources People who personally know who changed and what changed. They give initial authenticity, confidence and credibility to the information you harvest. Thus, Outcome Harvesting is highly participatory.

35 When? Who? What? Where? Less is more! 2. Review documentation and draft outcomes 3. Engage with informants to craft the outcome descriptions

Exercise Please take five minutes to read this 2012 annual report from FundNow. Then, with the person seated next to you: A. Identify one outcome: – Who changed behaviour, relationships, activities, actions, policies or practice? – What changed? – When did it change? – And where?

FundNow 2012 Annual Report Feedback from our allies and co-workers about FundNow’s activities during January-December 2012 has been very positive, emphasizing the usefulness of our research publications to inform about the funding challenges and opportunities facing women's organizations in the country. Our research has shown that women's rights organizations have been living in a form of "resist and survive" mode but doing amazing work for the advancement of women's rights. People always mention our online newsletter FundNow as a tool that has helped them to think more strategically about raising funds for their organizations. The preparatory research we did for each one of the13 meetings we convened was greatly appreciated. This year, our innovative research on trends in financing for gender equality and women's organization has been a strongly supportive tool for women’s groups obtaining more and better resources across the country. Based on the collection of "gems" of impact of projects implemented by the grant recipients of the government’s Fund for Leadership and Opportunities for Women (FLOW), we compiled a report that has been used by an ad hoc group of gender activists within the ministry responsible for FLOW. They used our compilation in their internal lobbying for the replenishment of FLOW, which resulted in the addition of US$1 million to the Fund in late In addition, this year we have successfully organized 2 strategic meetings for women's organizations on provincial resource mobilization. (In addition, we organised 7 more meetings in the capital for a total of 9.) There was intense interest in these meetings, the first held in those provinces. In one province, 5 funders and 18 women's organizations attended and in the other 4 and 12, respectively. The background research we presented was based on regional surveys carried out in preparation for the meetings; it was very well received. Participants said we were filling a clear gap of information with our findings. Many participants expressed that these meetings are a strategic opportunity to establish new contacts, engage in networking, develop strategies, and have a space for potential donors and grantees to meet and share experiences. Everyone said the meetings enabled them to better understand the context in which both funders and women's organizations operate. The meeting participants also had time to reflect on their work and said these events served as an inspiration. All the participants expressed their commitment to work together in the future to guarantee funding for women's organizations in the two provinces.

Exercise, continued B. How did FundNow contribute to the outcome? What activities and outputs plausibly contributed to the change in the social actor, however partial, indirect and even unintentional the contribution may have been?

Engage with Primary Intended User 1.To clarify and confirm the details of use 2.To establish the Outcome Harvesting questions to be answered 3.And identify the information (and its sources) required to answer the harvest questions 39

4. Substantiate: Verify accuracy + deepen understanding A. Description: In late 2012, the ministry responsible for FLOW took the decision to add US$1 million to the Fund. B. Contribution: FundNow compiled research that was used by the ad hoc group within the ministry to advocate internally for replenishment of the Fund. 1. To what degree are you in agreement with the description of the ministry’s action as described? [ ] Fully agree [ ] Partially agree [ ] Disagree Comments if you like: 2. How much do you agree with the description of how FundNow influenced the ministry’s decision? [ ] Fully agree [ ] Partially agree [ ] Disagree Comments if you like:

Triangulation of sources Historical documentation Person(s) who influenced others over time Independent third parties SOURCES

METHODS Triangulation of methods

5. Analyse and interpret the outcomes to answer the useful evaluation questions

Triangulation of analysis Methodologist Person(s) who influenced others over time Content experts ANALYSTS

Excel Database

Access Database

IMPACT Theory of change RESULTS SOCIAL ACTORS ASSUMPTIONS MISSION STRATEGIES

Systems approach

6. Support for use of findings Report And Support

In sum Outcome Harvesting is “an evaluation approach that does not measure progress towards predetermined outcomes, but rather collects evidence of what has been achieved, and works backward to determine whether and how the project or intervention contributed to the change.” - UNDP

Further information: Outcome Mapping Learning Community, Ricardo Wilson-Grau Consultoria em Gestão Empresarial Ltda Rua Marechal Marques Porto 2/402, Tijuca, Rio de Janeiro, CEP , Brasil Telephone: ; Skype: ricardowilsongrau Direct via VOIP, dialing locally from: The Netherlands (or Europe): USA or Canada: Questions?