1. Assessing the evidence base for how STIP enhances development programming
Annette N. Brown
Director, Research and Evaluation Strategic Initiative, FHI 360
Consultant, 3ie
@anbrowndc
American Evaluation Association
29 October 2016

2. 3ie scoping papers
Purpose: Identify priority questions for future impact evaluation and systematic review investment in a sector or for a theme
Inputs: Evidence gap map (supply of evidence) plus stakeholder consultation (demand for evidence)
Photo by Antonio Cinotti

3. Evidence gap maps
Many thanks for the invitation to present at the Foundation

4. David Halpern, CEO of UK’s Behavioral Insights Team, at What Works Global Summit
Governments should publish what they don’t know but need to know.

5. Objectives of 3ie evidence gap maps (EGMs)
To understand the size and scope of the evidence base for a given theme or sector.
To enable policy-makers and practitioners to identify and explore existing evidence for a given theme or sector.
To facilitate strategic and efficient research procurement.
Impact evaluation evidence gap maps can identify promising questions for synthesis research.
Impact evaluation evidence gap maps combined with information on evidence demand can identify priority questions for impact evaluation investment.

6. The basic EGM framework
Outcomes
Output 1
Outcome 1
Outcome 2
Outcome 3
Impact 1
Impact 2
Interventions
Interv 1
Interv 2
Study 1
Study 2
 Study 1
Interv 3
Study 3
Study 4
Study4
Interv 4
Interv 5

7. What EGM do not do
An EGM does not count, weight, or synthesize the evidence.
That is, more evidence does not necessarily mean evidence in favor of the intervention.

8. Preliminary results
Back in July we conducted our search – what did we get?

9. 297 2,034 Stage 3: Full text screening 9,086 31,616
37,494 Search results from all databases and websites
Stage 2: Abstract screening
Stage 1: Title screening
Duplicate removal (of databases)
Stage 4: Second full text screening and coding
727
291 impact evaluations complete ongoing
systematic reviews*
- 4 complete
- 2 protocol
297
Based on pre-specified search strategy, search of 12 databases and 35 websites – from that we yielded over 37k hits
Very exciting, much more than expected And still expecting more once snowballing is done – results are preliminary because coding much more than expected
Even with tech and innovation constraints found all this
Exciting news about state of evidence

10. Completed impact evaluation studies
Darker cells = more studies
Address gap problem (briefly explain cross-hatching)
Policies that affect innovation – global health
Looks like there may be a number of gaps
We will zoom in on some parts of the map

11. Completed impact evaluation studies
Darker cells = more studies
Address gap problem (briefly explain cross-hatching)
Policies that affect innovation – global health
Looks like there may be a number of gaps
We will zoom in on some parts of the map

12. Completed impact evaluation studies
Darker cells = more studies
Address gap problem (briefly explain cross-hatching)
Policies that affect innovation – global health
Looks like there may be a number of gaps
We will zoom in on some parts of the map

13. Evidence by intervention
Technology has almost double the number of categories than the other groups

14. Outcome type
Not surprising interventions can be randomized at ind. Level, or most studies use data from household or individual level surveys.

15. Sectors
Global health – we know from IER that majority of impact evaluations are health related

16. Technology interventions
To remind you, we had 191 impact evaluation studies in the technology group

17. Technology

18. Innovation ecosystems
Outcomes by level of analysis
LA1
LA2
LA3
Individual & household level outcomes
Organizational level outcomes
Community & societal outcomes
Intervention categories
Innovation ecosystems
Access to capital 4
Grants & subsidies 18
Policies & regulation that affect innovation 5
Networks & collaboration for innovation 3 1
Capacity building for innovation

19. Innovation ecosystems
Outcomes by area of focus
Education & Academic
Global health
Democracy, human rights & governance
Agriculture & food security
Crises & conflict
Economic growth, finance & trade
Environment & global climate change
Water & sanitation
Energy
Intervention categories
Innovation ecosystems
Access to capital 4
Grants & subsidies 18
Policies & regulation that affect innovation 5
Networks & collaboration for innovation 3 1
Capacity building for innovation

20. Evidence by country 234 completed studies Many in Asia – India, China
Latin America – Brazil, Argentina, Peru, Mexico
Gaps in mid east and north africa

21. Science interventions by country
Now we can look at the heat map for each intervention grouping

22. Technology interventions by country
Total number
192

23. Innovation ecosystems interventions by country
Government funded programs investing in econ growth

24. Partnership interventions by country
Haiti (1)
Rwanda (4)
Burundi (1)
Earliest/flagship world bank RBF program in Rwanda – all studies measure impacts of the same program
Indonesia (1)

25. Stakeholder survey

26. State of evidence for innovation intervention categories
- Access to capital the only category with less than 25% of respondents considering the evidence base to be weak

27. Preliminary findings
Need more evidence on impacts for community and societal outcomes
Need more evidence on the impacts of technology enhancements on agricultural and food security outcomes
Need more evidence on the impacts of technology enhancements on crisis & conflict and DRG outcomes
Need more evidence on the impacts of policies & regulations and networks for innovation ecosystems across all sectors
Need SRs for technology and education, innovation ecosystems and economic growth (esp org outcomes)