The ALNAP Meta-evaluation Tony Beck, Peter Wiles and John Lakeman

What is the ALNAP meta- evaluation?  An overview of evaluation of humanitarian action quality  Identification of strengths and weaknesses  Recommendations for improvement across the sector and in individual agencies

The ALNAP Quality Proforma ALNAP’s meta- evaluation tool Draws on good practice in EHA and evaluation in general Revised and peer reviewed this year

meta-evaluation Rated representative set of 30 evaluations Focus this year at request of ALNAP members on: Good practice Dialogue with 11 evaluation offices, focusing of impact of evaluation processes on evaluation quality

Agencies included in dialogue CAFOD, Danida, ECHO, ICRC, OCHA, OFDA, Oxfam, SC-UK, SIDA, UNHCR, and WHO

Findings from dialogue with evaluation managers Some areas affecting evaluation quality not currently captured by the QP Evaluation quality depends on subtle negotiations within agencies about key findings, eg staffing, use of DAC criteria Likely follow-up from recommendations is difficult to predict and dependent on a number of processes in agencies

Findings from dialogue with evaluation managers: the EHA market Main constraint to improved evaluation quality is agencies accessing available evaluators with appropriate skills Does the EHA market need further regulation?

Mainstreaming of the Quality Proforma By ECHO to revise tor (lesson learning, protection, identification of users, prioritisation, time frame and users of recommendations etc) DEC Southern Africa evaluation (rated 7 agency reports) Groupe URD (for planning of evaluations)

Findings from the Quality Proforma Significant improvement in use of DAC criteria, although efficiency and coherence still problematic Greater attention to protection (2002/3 – 6 per cent rates satisfactory or better, 2003/4 32 per cent rated satisfactory)

Findings from the Quality Proforma No improvement in appropriateness of evaluation methods use, vis a vis good practice Limited improvement in primary stakeholder consultation (13% satisfactory or better in 2002/3; 20% in 2003/4). Most other QP areas fairly similar to average Greater attention to HIV/AIDS

Good practice examples Follow-up Consultation with primary stakeholders Socio-economic analysis RTE Evaluation of efficiency Protection

Next steps Agencies valued interaction and dialogue with meta-evaluators, this should continue. Eg internal/external rating by agencies and meta-evaluators using slimmed down Quality Proforma (mainstreaming). Eg interaction between non-agency evaluators and meta-evaluators.

Next steps Is work needed on the EHA market, eg bringing in evaluators from the south Best format for proceeding – working group on evaluation quality?