1. EVIDENCE AID: The need for systematic reviews in the humanitarian sector and related areas TCD 2011 Bonnix Kayabu, MD Evidence Aid Coordinator Cochrane Collaboration & CGH TCD

3. Where are we coming from? Indian Ocean Tsunami (Dec 26 th 2004) Cochrane Collaboration and others to strengthen the use and the usefulness of SR Realise special collections of Cochrane Reviews Formal evaluation of Evidence Aid in 2008/9 (Turner 2009) Needs assessment of the use of systematic reviews in disasters settings and related areas

4. Reasons to use evidence (1) If an intervention is unsuccessful, the evidence should help to determine whether the intervention was inherently at fault or just badly delivered Secure resources for policy and account for how money is spent Relief agencies’ interventions tend to be ad hoc and would be more effective if they were based on the most up-to-date and available knowledge bases

5. Reasons to use evidence (2) Evidence will help us understand the risks, benefits and consequences of humanitarian choices. Evidence is most likely to correctly explain successes and failures of interventions (Bradt 2009) Good evidence can ameliorate or neutralise political obstacles (Gary Banks 2006)

6. Reasons for not using evidence (1) Evidence-based policy as an idea, but something that is rarely possible in reality (rhetorical device) Unfocused and contradictory evidence with lack of clarity about its robustness Very limited access to research evidence Research findings without take-home message (Lavis et al. 2004) “Many policy analyses occur behind closed doors and a political need for speed, or defence against opportunistic adversaries, are often behind that”. (Gary Banks 2006)

7. Reasons for not using evidence (2) Consultants or NGO Advisors with vested interest Increasing recognition of other factors Public health approach vs. Rights-based approach

8. Potential roles of Systematic Reviews Summarise the existing knowledge Show which interventions work, which don’t work, which are potentially harmful for people in need Research gap Mean decisions are not made based on the findings of a singular study, etc.

9. Purpose of the survey To identify the attitudes towards systematic reviews and research of those involved in humanitarian responses to disasters and other crises To identify their priorities for evidence To identify their preferences for ways to access the information

10. Quotes from earlier semi-structured interviews Poor Evidence “Evidence-based practice in humanitarian response is very poor. We do most things because we have been doing them year after year, we don’t do them because we have proven they are right. Very few have been really proven…” Problems with Guidelines “ … Most of the indicators are not measurable. And it doesn’t tell you how to measure them either. Even for the few, which are measurable, it doesn’t give any methodology…”

11. Quotes... Weaknesses in reporting system “It is important to start analysing the effectiveness of interventions, it is challenging but possible. The problem with aid in general is that it is made by a lot of anecdotal reports. There are more opinions than really hard data in NGO reports…” Problems with the quality of data “People using data from CRED sit in nice offices in London, Geneva or New York, they write their brilliant scholarly papers. They don’t understand just how unrepresentative this information is.”

12. Suggestions for priorities Effects of targeted supplementary feeding programs Political management of potable water Evaluation to ensure greater accountability for expenditure Reproductive, maternal and newborn health care focus in acute emergencies (it is often not prioritized) Culture norms (e.g. food preservation) Value of various interventions in lowering mortality Impact of training during emergencies Best practices in the use of social media in disaster response (e.g. for warning and evacuation), etc.

13. Rank these different types of evidence that might influence your decisions Anecdotal evidence 6.8% (4) Intuition 1.7% (1) Personal experience 11.9% (7) Scientific evidence 76.3% (45) Cultural norms 0.0% Organisation usual practice 3.4% (2)

14. How useful do you think SR can be in disaster settings? 81.4% SR are useful 18.6% Not sure if SR are useful No one said SR are not useful Action: increase access to reviews in these settings

15. Need for Reviews: Preliminary Results Have you used systematic reviews as a source of evidence in decision-making Yes : 52.5% No: 25.4% Not sure: 22.0% (N=59)

16. If you needed to access the findings of systematic reviews, how would you like them to be presented to you? Whole review: 45.1% (23) Whole review plus comments from experts in the humanitarian sector: 56.9% (29) Review summary on its own: 21.6% (11) Summary and context specific information: 47.1% (24) (N=59)

17. If you needed to access the findings of systematic reviews, how would you like to do this? [Answers] Full systematic review online: 82.2% (37) Summaries of systematic reviews online: 60% (27) Full systematic review by email: 35.5% (16) Summaries of systematic reviews on CD or DVD: 6.6% (3) Summaries of systematic reviews via mobile technology: 8.8% (4) (N=59)

18. Access to systematic reviews to improve responses to natural disasters

19. Donors want reviews Questions to donors: Would the use of systematic reviews help to assess the likely effects of projects before providing funding to agencies? Yes: 85% No: 10% I don’t know: 5% (N=20)

20. Conclusions Humanitarian aid workers are aware that evidence based practice in disasters is very poor Humanitarian aid workers and donors need systematic reviews to improve their interventions and assess the impact of their efforts They want reviews, comments from experts in the field and context-specific information to be accessed online or via mobile phone technology They have many uncertainties for which they need research evidence Evidence Aid should engage with aid workers to prioritise their needs on systematic reviews

21. Next steps for the survey Encourage more participants Comprehensive data analysis Key informant interviews Disseminate summary, interim reports Use the findings to inform the development of the future strategy for Evidence Aid

22. Priorities Identify new funding opportunities for EA Identify relevant SR Formal partnerships for EA Identify academic courses and provide training materials

23. THANK YOU!