1. Evaluating the system-wide effects of HIV scale-up: methodological gaps, challenges and recommendations David Hotchkiss Health Systems 20/20/Tulane University

2. Introduction  Wide agreement that further evidence is needed on how GHIs have influenced health systems  Research in this area has been characterized as serendipitous rather than systematic  Black box between inputs and outcomes, and intended and unintended effects, still needs illumination

3. Overview  What type of evidence is needed?  What are the obstacles to evaluation?  What are the available M&E approaches?  Where do we go from here?

4. What type of evidence is needed?  Routine monitoring  Monitoring of global goals and GHIs  Country-level health system and program monitoring focused on inputs, processes and outputs  Special purpose evaluations of the impact of health systems strengthening initiatives  All these aspects are overlapping and require an integrated approach at the country and global levels

5. What are the obstacles to evaluation?  Evaluations require collective action  Systems research traditionally has not received adequate financial support  High quality studies “do get done, but not in the numbers or with the quality that are justified by the global benefits” (CGD 2006)  Arrangements for prospective HSS assessments typically not built in from the beginning  Too little investment in baseline survey data  No treatment and control groups to assess counterfactual

6. What are the obstacles to evaluation? (2)  Routine tracking data should provide much of the data required for evaluation, but unfortunately, RHIS data are often incomplete and of poor quality  Attribution is challenging from a technical perspective  Systems strengthening a long-term, complex process  GHIs are also extremely complex and dynamic  Difficulty in attributing impact to any one GHI  Other factors, including concern about possible unfavorable results and limited capacity

7. What approaches and methods are available for monitoring and evaluation?

8. Example of a conceptual framework: IHP+ common M&E framework

9. What approaches and methods are available for monitoring?  National Health Accounts and Sub-Accounts, a tool for tracking sources and uses of funds  Comparison of NHA data over time can yield useful insights on system-wide effects  Health Systems Assessment Approach: A How-To Manual (Health Systems 20/20)  Provides indicators and sources of data by WHO’s health system functions

10. What approaches and methods are available for evaluation?  Retrospective  Global cross-country comparative analysis  Country-level mixed methods analysis (quantitative and qualitative, including case studies)  Prospective  Experimental designs assessing impact of interventions; economic evaluations of costs and benefits  Consider intervention-specific approaches  For some types of interventions, such as financing, service delivery, rigorous methods are available  For others, such as RHIS, more work is needed

11. Where do we go from here?  Great opportunity exists to invest in systems research  GFATM and GAVI increasing systems funding  G8 placing more emphasis on systems strengthening  IHP+ has developed a common evaluation approach  Develop “a new field of science for health outcomes and systems research among a constituency that has been fragmented in the past” (MPS 2008)

12. What does this involve?  Advocate rigorous M&E as a key component of the health systems strengthening agenda  In establishing HSS objectives of GHIs, establish systems to monitor and evaluate process  Align health systems M&E with implementation, country planning cycles and mechanisms  Continue to develop frameworks, metrics and methods  Invest in RHIS to strengthen data quality and use  Develop intervention-specific frameworks and methods  Continue to develop mixed-methods approaches  Look for opportunities for prospective evaluations

13. Thank you!

14. Selected references Atun RA, Bennett S, Duran A. When do Vertical (Stand-Alone) Programmes Have a Place in Health Systems? Policy Brief, WHO European Ministerial Conference on Health Systems, 25-27 June, 2008, Tallinn, Estonia. Banteyerga, H, Kidanu, A, Stillman, K. (2006). The Systemwide Effects of the Global Fund in Ethiopia: Final Study Report. Bethesda, MD: PHRplus. Abt Associates Inc. Global HIV/AIDS Initiative Network (GHIN) (2006). A Generic Guide to Research Practice: Following discussion at Lilongwe workshop of GHIN African teams. International Health Partnership (2008) Monitoring Performance and Evaluating Progress in the Scale Up for Better Health: A Proposed Common Framework. M&E Working Group. Kruck ME, Freedman LP (2008) Assessing Health System Performance in Developing Countries: A Review of the Literature. Health Policy 85. Rockefeller Foundation (2008) Leveraging HIV Scale-up to Strengthen Health Systems. Report of high-level meeting in Bellagio, Italy, 2-5 September WHO (2007) Everybody’s Business: Strengthening Health Systems to Improve Health Outcomes. WHO, 2007.

15. Selected references (2) WHO (2008) Maximizing Positive Synergies Between Health Systems and Global Health Initiatives. Report on the 3 rd expert consultation, WHO, Geneva, 2-3 October. WHO Maximizing Positive Synergies Collaborative Group (2009) An Assessment of Interactions Between Global Health Initiatives and Country Health Systems. The Lancet (373) June 20.