1. Assessing Humanitarian Performance: Where are we now? 24 th Biannual Meeting Berlin, 3 rd December 2008

2. 24th ALNAP Biannual Meeting, December 2008 2 Various strands of ALNAP work are working towards assessing system- wide performance  Various components of RHA - evaluation synthesis, meta-evaluation (especially on joint evaluations) and themed chapters  Facilitation of TEC and discussions on recommendations  HPP - data mapping and exploratory analysis of how to assess system-wide performance

3. 24th ALNAP Biannual Meeting, December 2008 3 What have we learned from the RHA?  Evaluation synthesis useful but, on its own, not able to assess performance  Component parts of RHA are good but the final product is probably less than the sum of its parts  Need to strengthen methodology and produce a more coherent whole

4. 24th ALNAP Biannual Meeting, December 2008 4 What have we learned from the TEC?  joint evaluations better than single-agency evaluations in providing system-wide snapshot  system-wide joint evaluations provide one off picture only  utilisation and take up of recommendations very difficult to achieve in practice

5. 24th ALNAP Biannual Meeting, December 2008 5 What have we learned from HPP?  Lots of data collected, but of different types, from different sources with different uses.  Many methodological and conceptual difficulties  Majority data gathered in needs assessment phase  Very little effort given to seeking the views of affected populations/ recipients of aid

6. 24th ALNAP Biannual Meeting, December 2008 6 What have we learned from the Madrid biannual? a)Be realistic as to what can be achieved now, use existing evidence to assess performance, with a special emphasis on ‘impact’ b)Explore the use of beneficiary surveys in assessment of impact and performance c)Develop a ‘pilot’ to test these ideas d)Continue mapping and do not lose sight of developing a more precise way of assessing performance

7. 24th ALNAP Biannual Meeting, December 2008 7 What are we going to do? The three-track approach:  Track one - fast track ‘State of the System’ pilot  Track two - medium track Learn more about use of beneficiary surveys and impact assessment and feed this into future ‘State of the System’ reports  Track three - slow track Continue mapping and work on developing key performance indicators

8. 24th ALNAP Biannual Meeting, December 2008 8 Track One: State of the System Report What is it for?  Overall goal is to assess overall humanitarian performance against agreed criteria  Pilot will provide a ‘base-line’ to track future performance

9. 24th ALNAP Biannual Meeting, December 2008 9 What problems will we face?  Analysing a system that is not strictly a system (i.e., not systematic)  Lack of data relating to outcomes and indicators

10. 24th ALNAP Biannual Meeting, December 2008 10 How will we address the problems? a) Break down the system into different units of analysis. Disaggregate the data by looking at: state of response in individual crisis state of response in particular sectors (clusters) state of response in particular categories – natural disasters, wars, high profile crisis, neglected crises state of response in relation to types of actors UN, NGOs, donors, governments etc

11. 24th ALNAP Biannual Meeting, December 2008 11 b) For each unit of analysis, performance will be analysed in relation to OECD-DAC criteria: Relevance/Appropriateness Connectedness Coherence Coverage Efficiency Effectiveness Impact

12. 24th ALNAP Biannual Meeting, December 2008 12 c) Need to identify indicators to apply to OECD- DAC criteria. For example: was coverage adequate: are resources adequate global funding against needs (CAP, beneficiary surveys) funding across sectors and emergencies staffing coverage in key areas and so on..

13. 24th ALNAP Biannual Meeting, December 2008 13 What do we want the report to tell us?  emerging themes  trends – how has sector y or response in emergency x changed over time  innovations and changes  performance indicators if/when they exist  perceptions of informed stakeholders about effectiveness, impact etc

14. 24th ALNAP Biannual Meeting, December 2008 14 What methods shall we use? Building on the RHA  key informant interviews (NGO’s, donors, government) aiming for mix of HQ and field  financial data analysis – (OECD-DAC and FTS)  mapping of global footprint and across current emergencies  key informant survey – polling opinion about performance to provide base line  evaluation synthesis  literature review

15. 24th ALNAP Biannual Meeting, December 2008 15 What now?  establish peer review advisory panel  undertake preliminary interviews/ consultations or input on scope and objectives of pilot  design detailed methodology and research plan in inception report to be peer reviewed