1. Land use change and forestry evidence gap map
Kristen Rankin
Evaluation Specialist, 3ie

2. Land use change and forestry - reconciling mitigation and food security: What do we know?
Agriculture, Forestry and Other Land Use sector (AFOLU) contributes approximately one quarter of all global GHG emissions.
The AFLOU is critical for global efforts to combat climate change. According to the most recent Intergovernmental Panel on Climate Change (IPCC) report the agriculture, forestry and other land use sector (AFOLU) contributes approximately one quarter of all global GHG emissions.
The main sources of these emissions are deforestation and agricultural emissions from livestock, soil and nutrient management, but forest degradation, forest fires and agricultural burning are also important (Smith et al. 2014). Emissions include both the release of CO2 (e.g., from deforestation or peatland drainage) and other GHGs such as CH4 (from livestock and rice cultivation) and N2O (from manure storage and agricultural soil and biomass burning). Because of the large share of GHG emissions from the AFOLU sector and the potential to provide carbon sinks by enhancing the uptake of carbon in soils and biomass it is vital for global efforts to curb climate change
Unique in that it can reduce emissions and enhance removal of gases.

3. Land use change and forestry - reconciling mitigation and food security: What do we know?
Close to 1.6 billion people – more than 25% of the world’s population – rely on forest resources for their livelihoods and many use trees on farms to generate food and income (FAO, 2015).
2.6 billion people depend directly on agriculture, providing livelihoods for 40 per cent of today’s global population.
At the same time the sector is essential to providing food for the global population of approximately 7 billion people and provides critical sources of fibre, fuel and other materials. Billions of people globally, especially in low and middle income countries (L&MICs), rely on agriculture and forestry for their livelihoods (Beddington et al. 2012). Land and forests also provide a multitude of ecosystem services, such as the provision of water, pest regulation, nutrient regulation, soil formation, and pollination that are fundamental to human well-being (Millennium Ecosystem Assessment 2005). The way in which agricultural and forest lands are managed is therefore critical for global efforts to achieve global food security, alleviate poverty and reach the Sustainable Development Goals (Beddington et al. 2012; Griggs 2013; Sachs 2012).
There is broad agreement that the AFOLU sector is critical for global efforts to mitigate climate change But there are also concerns that large-scale mitigation activities in the sector could have negative effects on food security, rural livelihoods and other aspects of human well-being. It is not clear which policies and programmes are most effective in reaching national and global emissions reduction targets, while avoiding negative effects on food security and other human wellbeing outcomes.
The Children’s Investment Fund Foundation (CIFF) is currently working on a programme to reconcile agricultural production and forest protection in Brazil. CIFF is working with 3ie to conduct primary research and evidence syntheses to support adjustment to activities as they are rolled out. To inform this work we conducted an EGM to take stock of the existing evidence on the effects of interventions in the sector.

4. Objectives of the 3ie/CIFF Evidence Gap Map
Identify, appraise and summarise existing evidence from systematic reviews of the effect of land-use change and forestry programmes on GHG emissions and human welfare outcomes
Identify existing evidence gaps where new primary studies and systematic reviews are needed to better inform climate change mitigation and development policies.
Key policy question: whether there are trade-offs or potential synergies between programme effects on environmental and human welfare outcomes, as measured by GHG emissions and food security.

5. What are Evidence Gap Maps?
3ie evidence gap maps are thematic collections of evidence in particular development sectors or thematic areas.
3ie gap maps include evidence on effectiveness of interventions, therefore contain collections of systematic reviews and or impact evaluations depending on the goal.
Collection is structured by a framework which schematically represents the key policy relevant interventions and key intermediate and final outcome categories, structured along the theory of change. Draw on systematic review and synthesis methodology.
ADD: Identify key gaps where we need more primary research and key clusters of evidence that have yet to be synthesized
3ie has developed a platform, an example of which you can see behind me, The colours indicate the confidence in the findings Green, yellow, red = high, medium, low confidence. Blue = protocol, Links to user-friendly summaries in the 3ie evidence databases. A tool for users to navigate the 3ie evidence portal-provides links to user friendly summaries

6. Evidence gap maps methodology
Develop Scope
Set Inclusion Criteria
Presentation and Analysis
Data Extraction and Critical Appraisal
Search for Relevant Studies
Assess for Inclusion
The evidence gap maps represent a graphical display of the number of existing Impact evaluations categorized under the interventions evaluated and their corresponding outcomes measured in a sector or subsector.
Develop Scope
Review key policy documents and other literature (including existing IEs)
Consult with key stakeholders including policymakers
Set Inclusion Criteria
If the purpose is knowledge translation, focus on systematic reviews
If literature is limited and the purpose is to inform research, EGM can also include primary studies
Search for Relevant Studies
Search drawing on systematic review methods, depending on time and resources available as well as the intended use of the EGM
Assess for Inclusion
Screen studies against substantive and study design inclusion criteria
Data Extraction and Critical Appraisal
Summarise included studies using a structured format and critically appraise SRs.
Code studies according to intervention/outcomes/status categories
Presentation and Analysis
Populate EGM with studies in appropriate cells
Time and resources permitting, produce a summary report with implications for policy and research.
Users should consider gap maps as one of a multitude of tools and sources of information to aid decision-making for policy and practice.
An EGM does not weight, or synthesize the evidence.
That is, more evidence does not necessarily mean evidence in favor of the intervention.

7. Land Use Change and Forestry EGM Scope
Area protection and management
Law & Policies related to forests and other land
Training, education and information
Incentives
Infrastructure
Human wellbeing and Environmental outcomes
Broad scope: Forestry and land-use (agriculture) programs or policies likely to have an effect on GHG emissions.
Area protection and management: Interventions in this category involve actions to establish or expand parks, reserves or other legally protected areas in which land or resource use is either fully restricted or regulated. They also include programmes with changes to the management regime of a particular area or jurisdiction.
Law & Policies related to forests and other land: actions to develop, change and implement formal legislation, regulations and voluntary standards related to forestry and other land.
Incentives: This category includes different interventions that use economic and other incentives provided to individuals or communities to influence land management behaviour.
Training, education and information: Interventions in this category are developed to promote sustainable practices and technology
Infrastructure: This category includes select infrastructure interventions which may have major implications for land use.
To limit the scope of the EGM and ensure included studies remain relevant to the focus of the EGM we applied additional inclusion criteria to studies of some interventions.
Specifically studies of certification, subsidies, grans and concessions, land rights, agricultural extension, technical and vocational training, information services, road construction, dams and infrastructure were only included if they fulfilled at least one of the following criteria: (1) promote sustainable agricultural practices, sustainable livestock, agroforestry, aquaculture, sustainable forest management, watershed management or a sustainable technology (eg: cookstoves, seeds, fertiliser, irrigation etc); (2) measure effects on an environmental outcome.
The following interventions were excluded: Social protection programmes; land use programs associated with construction, urban development and mining.

8. Land Use Change and Forestry EGM Scope
Study designs
Population / Setting
Outcomes
Impact evaluations: experimental or quasi-experimental designs
Systematic reviews of effectiveness
Any country
Included: Existing / degraded forests, grasslands, wetlands, mangroves, coastal or agricultural lands
Excluded: Marine, tundra or desert ecosystems or urban environments
Intermediate outcomes (e.g. uptake of a new practice)
Environmental outcomes (related to emissions)
Human welfare outcomes
We included impact evaluations meeting the 3ie IER inclusion criteria and systematic reviews of effects.
Studies which only investigate natural or market-based occurrences, or that report on the findings of controlled laboratory experiments or early-stage agricultural research station field trials with no discernable development intervention were excluded (3ie, n.d).
Geographical scope was global (expected that the evidence base would be limited in some intervention areas, could be relevant to LMICs)

9. 241 impact evaluations included 11 systematic reviews included
What did we find?
Full-text screening
criteria:
Date
Intervention
Outcome
Study design
Title screening criteria:
Intervention/relevance
68, 203
records identified through academic and grey literature searches
11,101 records screened at title (using priority screening)
1736 full-text articles assessed for eligibility
241 impact evaluations included
(11 ongoing)
11 systematic reviews included
(6 protocols)
241 included impact evaluations (+ 11 ongoing)
11 included completed systematic reviews (+ 6 SR protocols)
CAB Abstracts (Ovid), Greenfile (EBSCO), Econlit (Ovid), Scopus , 13 specialised topical databases and repositories of IEs

10. JUST ONE STUDY ON GHG EMISSIONS
Findings
OUTCOMES
LARGE BODIES OF EVIDENCE ON PROTECTED AREAS, PES, DECENTRALISED-FOREST MANAGEMENT
INTERVENTIONS
Intervention coverage (impact evaluations), main findings:
There is a fairly large evidence base of studies evaluating forest management: protected areas (56 included papers), payment for environmental services (41 papers), CFM/decentralised management (28 papers).
JUST ONE STUDY ON GHG EMISSIONS

11. Findings
OUTCOMES
SPARSE EVIDENCE ON TRAINING, INCENTIVES FOR SUSTAINABLE AGRICULTURAL AND LAND USE PRACTICES
INTERVENTIONS
Fewer papers evaluating sustainable agricultural interventions, both training and incentives, most measure intermediate outcomes such uptake of new practices, productivity of land.
HOWEVER - this is a fairly nebulous area, sustainable agricultural practices are not well defined within the sector and what is meant by sustainable can vary by context. Because of this we have been quite inclusive for this intervention area – for example we have included several studies related to fertiliser use from different countries, some promoting uptake of fertiliser in areas where soil fertility low and fertiliser use very low, and some encouraging reduction in fertiliser use.

12. Findings: outcomes measured in included impact evaluations
Social
96 studies
Environmental
103 studies 21
The vast majority of these evaluate intermediate environmental outcomes, mainly forest coverage, and do not go on to evaluate the ultimate environmental outcomes of interest (carbon storage, GHG emissions). Three studies reported on some measure of carbon storage or sequestration.
In terms of human welfare outcomes, income and household expenditure were measured in a large number of studies (n = 96). The studies used a diverse set of outcomes, typically income from specific livelihood sources, such as a specific agricultural product or from the forest, total household income or household expenditure. A small number of studies measured some type of poverty rate. Few studies assessed food security (n = 15), basic materials (n = 11) or health outcomes (n = 15).
Few papers evaluate both environmental and human welfare outcomes (the potential trade offs). A study of the REDD+ pilot programme in Nepal from 2015 measured carbon stocks in forests, in addition to human welfare outcomes. In addition, studies in Mexico, Cambodia of PES, protected areas in Costa Rica, Thailand, none looking at sustainable agriculture practices and human wellbeing.
VERY FEW STUDIES EVALUATE BOTH ENVIRONMENTAL AND SOCIAL OUTCOMES

13. Characteristics of the impact evaluation evidence
IMPACT EVALUATION LITERATURE DOMINATED BY QUASI-EXPERIMENTAL STUDIES, SOME RCTS IN AGRICULTURE
We found studies from 48 different countries. Latin America and the Caribbean and SSA has by far the largest number of studies.
The spread of studies across countries is uneven, with over half of the studies originating from only 10 countries (Costa Rica, Brazil, China, Indonesia, Mexico, Uganda, Ethiopia, India, Bolivia and Malawi). This is at least partially due to the existence of long-standing public forest protection programmes and availability of data in these countries. Across three of the most heavily studied intervention areas, namely PES, C/DFM and PAs, there is a concentration of studies in a relatively small number of countries. Forty-one studies evaluate PES, but these are concentrated in 14 countries, with 27 of the studies from just three countries: namely, Mexico, Costa Rica and China. In many cases there are a number of studies looking at the same programme, using the same or overlapping data sets.
The protected areas and payments for environmental services literatures are dominated by studies using PSM and other forms of matching. No RCTs of PES programmes for example – one ongoing by 3ie in Uganda.
Identified 18 included cluster RCTs, all from “sustainable” agriculture programmes (both extension and incentives encouraging use of fertilizer subsidies, IPM etc.).

14. Small systematic review evidence base
Moving to the systematic review base, we found 11 systematic reviews. 5 rated high confidence.
Five of these reviews synthesised evidence of DFM or CFM interventions, three systematic reviews assessed the evidence on effects of PA on forest coverage, and human wellbeing outcomes (Pullin et al. 2013), while two systematic reviews assessed the effects of PES. A further two reviews covered some kind of multicomponent intervention (Brooks et al. 2013; Roe et al. 2015). Two systematic reviews focused on agriculture but included environmental outcomes, namely agricultural extension and training (specifically farmer field schools; Waddington et al. 2014) and agricultural subsidies, grants and concessions (specifically agricultural innovation grants for farmers; Ton et al. 2013). Finally, one systematic review by Lawry et al. (2014) looked at the impact of tenure formalisation.
We also found six relevant ongoing systematic reviews (where a protocol is publically available) in the map. These focus on silvicultural interventions, certification for improved agricultural production, property rights regimes, China’s conversion of cropland to forest programme, CFM and long-fallow swidden agriculture systems versus alternative land uses and livelihoods and carbon storage in South East Asia.

15. Implications for research: update existing and undertake new systematic reviews
Update high confidence but out-of-date reviews:
Payment for Environmental Services (PES)
C/DFM
Land property rights
New reviews:
Protected areas and environmental and human welfare outcomes
Agricultural extension and training– “sustainable agriculture”
Most of these high confidence reviews are now out of date…we can see from looking at the impact evaluation base that a number of studies have been published since the high quality reviews were undertaken, particularly evaluating protected areas and PES. Great opportunity to update these reviews.
Specifically, there is a need for new, updated syntheses of the effects of PA, PES, C/DFM and agricultural extension and training. For example, Pullin et al. (2013) synthesised the effects of PAs on human wellbeing, but this review is now several years old, with at least four new studies published since the research was conducted. Similarly, there are several systematic reviews of PES and C/DFM assessing effects on both environmental and socioeconomic outcomes. However, the research of the most recent review (Samii et al. 2014a) was conducted in 2012, and at least six studies have been published since then, suggesting that an update may be warranted.
While systematic reviews exist for some of these interventions, they are either out of date and/or have major weaknesses which reduce confidence in their findings.
The number of studies assessing the effects of agricultural extension and training suggest a systematic review focusing on these interventions could add value. As noted above, these studies cover a diverse range of practices and technologies that are either described as sustainable, or that may influence environmental outcomes. The specific practices and technologies promoted for scale-up are likely to differ somewhat between contexts, but a systematic review assessing the effectiveness of strategies or mechanisms for promoting farmer adoption of new practices and technologies (as well as the impacts of adoption on GHG and human wellbeing) could inform the design of future programmes.
Finally, lack of clarity on practices and technologies with climate change mitigation potential highlights the need for systematic reviews to assess the efficacy of different approaches promoted as ‘sustainable’ or ‘climate smart’. It was beyond the scope of this study to map this literature so we are not able to say anything about the scale of the underlying evidence base. However, reviewing such studies could help identify efficacious practices and technologies to be promoted for take up by farmers. Rosenstock et al. (2016) is an example of an ongoing systematic review looking at parts of this literature. There may be others, and any future syntheses should start by taking stock of related ongoing and completed systematic reviews assessing the efficacy of ‘sustainable’ or ‘climate-smart’ technologies and practices.

16. Implications for research: new primary studies
More studies to assess the potential trade offs of climate mitigation - effects on human welfare outcomes, particularly food security, AND environmental outcomes (GHG emissions and/or carbon storage
Exploit opportunities for randomisation or high quality quasi-experimental studies
Work on testing ‘sustainable agriculture’:
efficacy of technologies in different land systems
programmes for dissemination and evaluate

17. Conclusions
The AFOLU sector is critical for global efforts to mitigate climate change and provide livelihoods and food security for billions
Urgent need for coordinated effort and funding for new studies that consider both emissions / carbon stocks and human wellbeing outcomes of different land and forestry interventions

18. And finally…
Authors: Birte Snilstveit, Jennifer Stevenson, Paul Fenton Villar, John Eyers, Celia Harvey, Steven Panfil, Jyotsna Puri, Madeleine C McKinnon
Report available at:
Evidence gap map available at:
Queries:

19. Systematic review evidence library:
Evidence gap map library:

20. Appendix
Kristen

21. There has been an increase in efforts to produce evidence on what works, including both systematic reviews and high quality impact evaluations.
Research reports are often long and technical, scattered across websites.
Many of these reports are never read. But they may have nuggets of knowledge or important findings.
A WB report analysing use of their reports found that 1/3 of reports had never been downloaded.
Only 40% had been downloaded less than 100 times
Only 13% downloaded more than 250 times.
87 % of policy reports never cited.
How can we ensure evidence is presented in a useful and accessible format?
How can we ensure limited resources are spent efficiently and important evidence gaps prioritised?

22. EGMs: Supporting evidence-informed policy making
Outcomes
Interventions
1. Policy process moves quickly and PM do not always have time to wait for new IEs and SRs to be completed
EGMs can be conducted relatively quickly and aim to ensure best existing evidence is available when PM need it.
2. Reduce wasted opportunities to improve outcomes because evidence is unavailable. Highlight high quality studies, so PM can make informed judgement about what research to rely on.
3. Overview of existing SRs on a topic, with links to user friendly summaries on SR database.
- Enable policy-makers and practitioners to explore findings and quality of existing evidence on a topic.
- When used to inform policy should draw on SRs.
If evidence base relatively mature, can allow for comparisons of relative effectiveness.

23. Impact evaluations and consideration of equity in the AFOLU sector
195 impact evaluations did not consider equity
Gender
Socio-economic status
Place of residence
Race, ethnicity, culture, language
We aimed to identify how and to what extent the included studies incorporate equity in their assessment of the intervention. That is, does the paper assess the extent to which the intervention affects populations or specific groups in different ways.
Figure 6 shows the distribution of impact evaluations by consideration of equity. A large majority of the studies do not consider equity in any way (n = 195), either by studying an intervention aiming to reduce inequality, targeting specific groups or by assessing effects on different groups.
In addition, 18 studies assessed the effect of an intervention targeting a specific group, such as socioeconomic status (n = 13), women (n = 5) or place of residence (n = 3).
Twenty-one studies conducted subgroup analysis for one or more potentially vulnerable groups. Of these studies, 13 assessed effects by gender, 10 by socioeconomic status, 3 by educational status, 3 by age, 2 by race, ethnicity, culture or language (indigenous groups) and 1 by place of residence.
Finally, seven papers assessed the effect of interventions aimed at reducing inequality.

24. Decentralised forest management Payments for environmental services
Implications for policy: findings of the high confidence systematic reviews
Decentralised forest management
(Samii et al., 2014a)
Positive, though modest, effects of DFM
Unable to rule out a negative effect on poverty
Payments for environmental services
(Samii et al., 2014b)
Modest reductions in deforestation, but inefficient
Two studies suggested modest improvements in household income
In 2014, two reviews published by Campbell on decentralised forest management and payments for environmental services, looking at both deforestation and poverty outcomes, exploring the trade offs between the two.
The DFM review finds some evidence for positive, though modest, effects of DFM on deforestation. The authors are unable to rule out a negative effect of DFM on poverty. The authors highlight the lack of high quality studies assessing the effects of DFM on environmental and human welfare outcomes. They encourage researchers to take advantage of the opportunity for randomisation provided by the phasing in of local DFM establishment programmes. They also recommend researchers replicate the quasi-experimental approaches employed in some of the studies included in the review and make use of tools such as Google Earth Engine’s high resolution forest cover mapping for formative research that can inform more finely targeted experimental studies.
Samii et al. (2014b), found that PES reduce deforestation, but impacts are modest and inefficient, reducing the annual deforestation rate by 0.21 percentage points on average. Two studies suggested modest improvements in household income. The findings come from 11 studies that evaluate the effects of six different PES programmes in four different countries. The authors concluded the evidence base on the effects of PES programmes is limited in both quantity and quality

25. Protected areas and human well-being
Implications for policy: findings of the high confidence systematic reviews
Protected areas and human well-being
(Pullin et al., 2013)
Rigorous impact evaluation evidence disparate and fragmented
Existing evidence base is insufficient to draw conclusions about the scale of positive or negative impacts of PAs
Land property rights
(Lawry et al., 2014)
Improved land property rights improved tree crop planting in Ethiopia, Nicaragua and Vietnam
Improvement of around 15 per cent on average for human welfare outcomes
One high-confidence review examined effects of terrestrial PA on human wellbeing (Pullin et al. 2013). Despite being the most frequently used conservation interventions, the authors find that rigorous impact evaluation evidence remains disparate and fragmented. The existing evidence base is insufficient to draw conclusions about the scale of positive or negative impacts of PAs. Impacts of Pas are highly context dependent and the quality and quantity of rigorous evidence restrict their ability to generalise any policy recommendations.
Lawry et al. (2014), a high-confidence review, found that land property rights improved tree crop planting in Ethiopia, Nicaragua and Vietnam (Lawry et al. 2014). Agricultural productivity also improved in some contexts, with gains being more limited in Africa than other regions. The evidence suggested an improvement of around 15 per cent on average for human welfare outcomes, as measured by income or consumption.