1. Presentation of Final Report The Results of the Thematic Evaluation on Maternal Health (2000-2011) United Nations Population Fund Evaluation Branch Division for Oversight Services New York, 07.11.2012

2. Overview of proceedings 10:30Introduction: Objectives, scope of evaluation Methodology used Louis Charpentier, Evaluation Manager, Evaluation Branch / DOS (UNFPA) 10:40Presentation of key findings and conclusions of thematic evaluation Martin Steinmeyer, Team Leader (AGEG) Isabelle Cazottes, Co-Team leader (AGEG) 11:20Open discussion 11:40Presentation of the recommendations of the evaluation Martin Steinmeyer 12:00Use of evaluation resultsLaura Laski, Chief – Sexual and Reproductive Health Branch, Technical Division (UNFPA) 12:30Open discussion 12:50Next stepsLouis Charpentier 2Evaluation Branch, Division for Oversight Services

3. Evaluation Manager Chief, Evaluation Branch, Division for Oversight Services Introduction: Objectives, Scope and Methodology of the Thematic Evaluation 3Evaluation Branch, Division for Oversight Services

4. Objective, users, rationale of maternal health thematic evaluation Objective Assess relevance, effectiveness, efficiency and sustainable of UNFPA in contributing to improvement of maternal health Evaluation Users UNFPA country offices, Programme and Technical Divisions, including MHTF staff Partner Governments, member states, donors, civil society Rationale & Purpose Inform decision-making on policy / project management Establish accountability and oversight of fund management Promote lesson-learning culture Provide guidance on strengthening partnerships 4Evaluation Branch, Division for Oversight Services

5. Thematic & temporal scope MYFF 2000 - 2003MYFF 2004 - 2007Strategic Plan 2008 - 11 2000 2010/11 Capacity Development / Human Resources for Health SRH Services (Family Planning / EmONC) Results & evidence orientation Harmonization & coordination of support Coherence of SRH, Population & Development, Gender Relevance of UNFPA support Integrating MH into national policies & frameworks Coherence between country, regional & global programmes Visibility of UNFPA in maternal health MH support in humanitarian contexts 5Evaluation Branch, Division for Oversight Services

6. Geographic scope of evaluation 1 st stage sampling (“Universe”) 55 programme countries with MMR greater than 300 death / 100,000 life births 2 nd sample 22 countries (range of MH performance, GNP, quality of public admin, HIV prevalence) 3 rd sample (country case studies) Burkina Faso, Cambodia, DRC, Ethiopia, Ghana, Kenya, Lao PDR, Madagascar, Sudan and Zambia 6Evaluation Branch, Division for Oversight Services

7. Analysis and reconstruction of UNFPA’s own intervention logic (Core fund (& MHTF)) Analysis of the context of UNFPA support to maternal health Identification of critical aspects (Core Fund & MHTF) DAC Criteria (Relevance, Effectiveness, Efficiency, Sustainability) 2 sets of Evaluation Questions: 1.Maternal Health Thematic Evaluation (12 EQs) 2.MHTF Mid-Term Evaluation (8 EQs) UNFPA results frameworks (MYFFs, Strategic Plan, etc.) MHTF Business Plan, etc. Analysis of strategies of other UN agencies & donors Academic and “gray” literature on reproductive & maternal health ToR / Input from Reference Group Main Topics / Issues / Purpose identified in ToR; specified by Reference Group Methodology – Definition of Scope 7Evaluation Branch, Division for Oversight Services

8. Methodology – Data Collection  Data were collected to answer 12 Evaluation Question of Thematic Evaluation  Data collected by following means: Desk review of existing evaluations, reviews and other documents; Ten country case studies (Burkina Faso, Cambodia, Ethiopia, Ghana, Lao PDR, Madagascar, Sudan (North) and Zambia); focus groups, interviews with UNFPA staff, Government, development partners, civil society, beneficiaries Online survey of 55 country offices on technical support to country offices & on organizational capacity Face-to-face and telephone interviews with UNFPA staff members (headquarters, regional and sub-regional offices), other external partners. 8Evaluation Branch, Division for Oversight Services

9. Methodology – Number of People Interviewed Institutions Number of People Interviewed UNFPA23 (global), 6 (regional), 65 (country) Central Government90 Local Authorities50 Development Partners8 (global) 65 (country) Civil Society Organisations55 Training Institutions30 Health Service Providers40 Final Beneficiaries (focus group discussions) 18 discussions (10-15 people each) 9Evaluation Branch, Division for Oversight Services

10. Martin Steinmeyer, team leader (AGEG) Isabelle Cazottes, co-team leader (AGEG) Key Findings and Conclusions: The UNFPA contributions to changes in maternal health outcomes 10Evaluation Branch, Division for Oversight Services

11. SELECTED KEY FINDINGS Answers to the Evaluation Questions 11Evaluation Branch, Division for Oversight Services

12. Evaluation Question 1: Relevance  Resources not allocated proportional to severity of maternal health needs Resource allocation system has not fully considered relative need levels in each country Resources distributed based on criteria such as “degree of political support for ICPD”, “absorptive capacity”  No clear, operational definition of maternal health-related vulnerability Common practice: targeting MH support geographically (regions / districts with high maternal mortality prevalence) Less common: targeting systemic socio-economic barriers to MH services (cost, distance, transportation) 12Evaluation Branch, Division for Oversight Services

13. Evaluation Questions 4: Capacity Development – Human Resources for Health  Training support thematic priority of UNFPA 2000 – 2005: in-service trainings; Challenges: Low alignment with national HRH systems; Little discernible sustained effect on quality of service 2005 – 2010: pre-service trainings, support of nat. training oversight agencies (curricula review, regulatory frameworks); Advantages: alignment w. nat. structures, sustained effects on SRH training systems  Challenge: Inappropriate deployment & placement of trained staff / high mobility Linked to weak nat. HRH systems (planning, implementation, monitoring) – beyond SRH / MH UNFPA offices struggled to approach cross-cutting challenges Some partnerships w. HRH offices in health ministries (e.g., Cambodia) 13Evaluation Branch, Division for Oversight Services

14. Evaluation Question 7 – Emergency Obstetric and Newborn Care (EmONC)  EmONC support prior to 2008, but MHTF has intensified UNFPA EmONC support Technical assistances, finances sped up EmONC needs assessments; basis for scale-up plans (Lao PDR)  But: Availability of; access to EmONC affected by admin. gaps, cost, weak referral systems UNFPA offices struggled to approach cross-cutting challenges (barriers) (UNFPA mandate? Capacity?)  Positive exceptions: 1) Analysis of barriers; 2) Long-term Partnerships in- & outside SRH; 3) Advocacy Burkina Faso: EmONC subsidy, emerged out of long-term UNFPA advocacy, partnership w. Direction de La Famille (MoH), Parliamentarians; increase in facility-based deliveries (28% overall (2003 - 2010)) 14Evaluation Branch, Division for Oversight Services

15. Overview of key findings for other evaluation questions (selection) QuestionsKey Findings EQ2: Harmonization / Coordination Long-term partnerships prerequisites for UNFPA effect on evidence-based harmonized MH support EQ3: Communities & MH demand MH awareness raising to be coupled with addressing financial barriers to be successful EQ6: Family planning Good results / partnerships in commodity security (GPRHCS); less focus on FP demand / delivery EQ8: Results / Evidence Weak monitoring and evaluation of MH interventions; low results-focus EQ9: MH policy frameworks Success depended on combining data & research, advocacy, technical assistance, partnerships EQ10: SRH, Gender, P&D Country offices commonly lacked management mechanisms for integrated programming (strategy!) EQ12: VisibilityVisible advocate for MH; extent depended on CO capacity to bring technical knowledge to bear 15Evaluation Branch, Division for Oversight Services

16. Performance patterns of country-level support – Example HRH CO Management P&D SRH Gender RH policy arena Health policy arena (& other policy arenas) SBA assessm. TA SBA Policy advocacy SBA plan; midwifery training policy HRH Policies / Programmes Long-term strategy HRH Dep. / MoH MH Centre / Nursing Councils E.g, poor deployment E.g, WHO Ohter donors Strengthening deployment system Curriculum reviews; Training 16Evaluation Branch, Division for Oversight Services

17. Performance patterns of country-level support - positive scenario Management P&D SRH Gender RH policy arena Health policy arena (& other policy arenas) Data / research Technical Assistance Policy advocacy MH policies & programmes Health Policies / Programmes Long-term strategy Gov. partner MH Barriers Dev. partner System Strengthening Implement. Support Country Office 17Evaluation Branch, Division for Oversight Services

18. Performance patterns of country-level support – opportunities for improvement 18 CO Management P&D SRH Gender RH policy arena Health policy arena (& other policy arenas) Data / research Technical Assistance Policy advocacy MH policies & programmes Health Policies / Programmes Short-term planning (project by project) Gov. partner MH Barriers Dev. partner Curriculum reviews; Training Country Office 18Evaluation Branch, Division for Oversight Services

19. Conclusions

20. Conclusions - Appropriateness of UNFPA maternal health strategy C1: UNFPA maternal health support in programme countries has not been sufficiently based on country-specific medium or long-term strategies  Maternal health support has been more effective when country offices based interventions on multi-annual strategic vision  Current templates not conducive for strategic planning; No requirement to develop multi-annual operational plan.  Country offices without multi-annual perspective more likely to manage interventions separately from each other;  Fewer incentives for staff working on different interventions to coordinate work; pool financial and organizational resources from different funding sources 20Evaluation Branch, Division for Oversight Services

21. Conclusions - Appropriateness of UNFPA maternal health strategy (2) C2: In its approach to support maternal health, UNFPA has not sufficiently responded to its mandate to focus on addressing the root causes of poor maternal health of the most vulnerable  HQ has not sufficiently defined operational implications of focusing on needs of “most vulnerable”  Country offices did not sufficiently analyse risks and barriers keeping women and girls from accessing MH services (instead targeted geographical pockets of high maternal mortality)  COs did not address weakness of health systems that made specific groups more vulnerable to poor maternal health (e.g., deployment challenges, in particular for rural areas; deficiencies in supervision and service quality control) 21Evaluation Branch, Division for Oversight Services

22. Conclusions - Appropriateness of UNFPA maternal health strategy (3) C3: UNFPA support of maternal health services at sub- national level has not consistently reflected comparative strengths of UNFPA as primarily knowledge- and evidence-based organization  Significant portion of small SRH budget allocated to sub- national level (sub-national offices, staff, interventions); often one of several organisations active there  Country offices have not consistently used presence at sub- national level to generate data, lessons and knowledge to further national MH policy agenda 22Evaluation Branch, Division for Oversight Services

23. Conclusions – role & capacity of UNFPA country offices C4: Insufficient staff capacity and skill gaps have negatively affected ability of country offices to act as brokers of maternal health expertise and as facilitator of national and international maternal health commitments and partnerships  Small numbers of RH staff made it difficult to be present in national technical working groups or policy forums  Annual planning based on inadequate templates (AWPs, CPAPs) and limited experience in strategic planning made it difficult to develop multi-annual strategies that combined resources and skills from different areas to facilitate sustained changes in health systems 23Evaluation Branch, Division for Oversight Services

24. Conclusions – role & capacity of UNFPA country offices (2) C5: Country offices have not received sufficient technical support from regional offices and headquarters to fulfill central role delivering maternal health support  Availability of technical support has been limited in human resources for health, EmONC and midwifery (pre-MHTF), strategic planning, results-based management and monitoring and evaluation  Maternal Health Thematic Fund (MHTF) has been valuable instrument to make available additional resources for technical support 24Evaluation Branch, Division for Oversight Services

25. Conclusions – partnerships with donors, governments and other stakeholders C6: Varying capacity of country offices to establish partnerships with government, donors and civil society in and outside of reproductive health has reduced ability to address service access barriers and to strengthen MH systems  Long-term, proven working partnerships allow UNFPA to extend reach beyond sexual and reproductive health Partnerships allow UNFPA to tie maternal health HRH support to larger governmental efforts to strengthen national health systems  Pre-conditions: UNFPA to repeatedly prove worth as reliable partner, gain trust and collaboration, consistent leadership through country representatives 25Evaluation Branch, Division for Oversight Services

26. Conclusions – Use of Evidence; Monitoring and Evaluation C7: Lack of appropriate monitoring and evaluation mechanisms has affected capacity of UNFPA to assess results of maternal health support and to optimize corporate and country-level MH strategies over time.  Monitoring has focused on activities or higher level societal changes relating to maternal health,  Monitoring system thus not providing information on UNFPA contribution to these changes (which mirrors deficiencies in UNFPA planning)  UNFPA implementing partners w/o required technical capacity to fulfil M&E responsibilities; UNFPA country offices w/o sufficient skilled M&E staff to support implementing partners 26Evaluation Branch, Division for Oversight Services

27. Conclusions – Added value of the Maternal Health Thematic Fund (MHTF) C8: Maternal Health Thematic Fund (MHTF) has helped to provide much needed financial and staff resources to UNFPA country offices and headquarters in the short and medium-term.  MHTF-financed staff positions bolstered staff capacity of country offices, allowing them to intensify engagement in in EmONC, midwifery, obstetric fistula  Partnerships made available additional technical support (e.g., Averting Maternal Death and Disability (AMDD), International Confederation of Midwives (ICM))  Helped to raise UNFPA profile and visibility globally & in countries 27Evaluation Branch, Division for Oversight Services

28. Conclusions – Added value of the Maternal Health Thematic Fund (MHTF) C9: The MHTF has not been sufficiently integrated into organizational structure of UNFPA and the overall planning process at country level to ensure sustainability of its interventions  MHTF has proved instrumental in facilitating EmONC assessments, midwifery needs assessments, which constituted basis for governments to launch MH-relevant reforms  MHTF not sufficiently integrated into UNFPA planning to ensure timely resource mobilization from within UNFPA and partners for continued support of reforms 28Evaluation Branch, Division for Oversight Services

29. QUESTIONS? COMMENTS? 29Evaluation Branch, Division for Oversight Services

30. Recommendations

31. Recommendations: Reviewing UNFPA maternal health strategy and support R1: UNFPA should revise internal procedures, tools and templates for strategic planning. New approach should require country offices to develop maternal health support strategies for medium to long-term, and to detail how resources from different sub-programmes will be used to implement these strategies. (based on conclusions C1, C7)  Present detailed analyses of specific political, administrative, socio-economic challenges in revised CPDs and CPAPs and corresponding strategy to address them  Put in place multi-annual plans that detail theory of change, risks & assumptions and monitoring indicators for strategy 31Evaluation Branch, Division for Oversight Services

32. Recommendations: Reviewing UNFPA maternal health strategy and support (2) R2: UNFPA needs to better define operational implications of targeting the needs of the “most vulnerable”. Concept is relevant part of maternal health strategy, yet it is too vague in its current form to guide maternal health programming at country level. (based on conclusion C2)  Develop typology of MH-relevant barriers for different types of services and contexts  Prepare operational and programming guide to a) offer clear definition of MH vulnerability and the contributing social, economic, political variables; b) identify policy sectors that influence MH vulnerability; c) lay out options to influence determinants of vulnerability; d) define UNFPA role in contributing to donor response; e) discuss use of partnerships 32Evaluation Branch, Division for Oversight Services

33. Recommendations: Reviewing UNFPA maternal health strategy and support R3: UNFPA needs to increase focus on knowledge generation and learning, to make use of lessons from sub-national service delivery support (and other interventions) to inform evidence-based policy advocacy at core of UNFPA mandate. (based on conclusion C3, C7)  Service-delivery support needs to contribute to body of knowledge on maternal health support in programme country and beyond  Strengthen provisions for monitoring, evaluation, learning, in particular for interventions at sub-national level 33Evaluation Branch, Division for Oversight Services

34. Recommendation: Improving capacity of country offices R4: UNFPA needs to better align capacity and skill mix of staff and managers to ensure that country offices can fulfill their role as knowledge brokers and facilitators of evidence- based approaches to improve maternal health. (based on conclusions C1, C4, C7)  Country offices to develop resource plan (part of CPAP) to allocate staff time to strategy components  Strengthen skills related to health policy and management, public health ; also project management, M&E  Strengthen accountability of country representative for strategic positioning of country offices for partnerships, advocacy to complement UNFPA interventions 34Evaluation Branch, Division for Oversight Services

35. Recommendation: Improving capacity of country offices R5: Planning process for regional-level technical support to be better aligned with long-term strategic and operational planning for maternal health support at country level. (based on conclusions C4, C5)  Long-term strategic planning at country level needs to be mirrored by long-term planning of technical support at regional level  Regional offices’ planning processes need to address current country-level programming needs and anticipate future support requirements  RPAPs / Country Programmes to be developed jointly with country offices, HQ, MHTF, GPRHCS, including resource plan (similar to country level) 35Evaluation Branch, Division for Oversight Services

36. Recommendations: Improving guidance on UNFPA partnerships R6: Anchor concept of partnerships more firmly in strategic documents, operational guidelines and job descriptions of managers. Strategic documents need to explain importance of different types partnerships to ensure sustainable results. UNFPA managers need to be responsible for setting up required partnerships at country, regional and global levels. (based on conclusions C1, C6)  Develop typology of stakeholders and their significance for addressing root causes of poor maternal health  Strengthen capacity of regional offices to support country offices in establishing partnerships  Job descriptions of managers need to emphasize their responsibility for strategic positioning of UNFPA through partnerships (performance criterion) 36Evaluation Branch, Division for Oversight Services

37. Recommendations: Improving UNFPA provisions for monitoring R7: Strengthen result-oriented monitoring for country offices to measure results and not only activities and inputs. Assign greater responsibilities and offer more guidance to country offices for supporting the set-up of appropriate monitoring mechanisms with implementing partners. (based on conclusions C1, C4, C5, C7)  Develop guidelines for results-oriented monitoring & related training resources  Strengthen responsibilities and capacity of country offices to support set-up of monitoring mechanisms with implementing partners (consider partnerships) 37Evaluation Branch, Division for Oversight Services

38. Recommendations: Revise role & integration of MHTF R8: Strengthen MHTF as catalytic tool to facilitate implementation of evidence-based maternal health interventions. Clarify that MHTF is not only responsible for launching initiatives (EmONC, midwifery), but also for assisting country offices to support these initiatives until completion. (based on conclusions C8, C9 + MHTF-mid- term evaluation)  Clarify: MHTF helps launch initiatives; and supports their implementation (business plan)  Update guidance to country office on purpose and role of MHTF Support to expect from MHTF Responsibilities of country offices when working with MHTF funds  Harmonize MHTF planning with (proposed) multi-annual planning approach for core funds 38Evaluation Branch, Division for Oversight Services

39. Recommendations: Revise role & integration of MHTF R9: UNFPA should use MHTF funds to carry out pilot interventions on selected core maternal health issues, such as the development of appropriate support strategies to better target populations with high vulnerability to poor maternal health. (based on conclusions C3, C8, C9)  Inventory of MH issues and topics that require pilot-testing  Invite country offices to submit proposals for pilots for maternal health interventions  Ensure access of country offices to required resources for pilots through MHTF  Ensure analysis and dissemination of results with MHTF resources 39Evaluation Branch, Division for Oversight Services

40. QUESTIONS? COMMENTS? 40Evaluation Branch, Division for Oversight Services

41. NEXT STEPS - Presentation of the evaluation results and the management response to the Executive Board – January 2013 - Webinar presentation of the management response – February 2013 41Evaluation Branch, Division for Oversight Services

42. Maternal Health Thematic Evaluation Report and related products are available at the following link: 42Evaluation Branch, Division for Oversight Services http://www.unfpa.org/public/home/ about/Evaluation/EBIER/TE/pid/10094

43. THANK YOU FOR YOUR ATTENTION 43Evaluation Branch, Division for Oversight Services Any other questions? Contact us: evb@unfpa.orgevb@unfpa.org