1. Monitoring health system performance - s ynthesis of some experiences from low-income countries Dina Balabanova, Tim Powell-Jackson, Richard Coker, Kara Hanson & Anne Mills London School of Hygiene and Tropical Medicine Health System Metrics, Glion sur Montreux, 28-29 September 2006

2. Overview n Background n Complexity n Objectives and methods n Measurement n Health financing n Health care delivery n Emerging issues n Conclusions

3. Background n Commitment to invest in health systems is unprecedented, but will not last unless it is possible to show results n Currently poor health information available but demand for improved health system metrics (national / international) n Opportunities –Health System Metrics and other initiative seeking to strengthen HIS –Commitment to the health MDGs – need to measure progress –Growing consensus of importance of measurement strategies & monitoring & evaluation built into programme planning cycles n Threats –Limited resources for health information and sustainability –Capacity constraints (in the health and social sectors)

4. Objectives & Methods n Purpose of the study: a review of some low-income countries experiences with health system performance monitoring and use of data n Case study countries: –Georgia –Rwanda –Uganda –West Bengal, India –Material from other countries n Selection criteria n Analytical approaches: –uses the WHO health system performance framework –synthesis around common themes and issues –identifying unique lessons in each type of context

5. Complexity n How should health system performance be measured? –Increasingly multiple contacts with the system, chronic diseases –Outcomes determined by different care components, sectors –Need for system-wide and inter-sectoral indicators n Tension between international (donor-driven) demands and country-level agendas and needs n Use of normative approaches imply causality n To what extent monitoring influences policy? n Impact of measurement on health systems, e.g. Indicators that are measured often improve n Monitoring information may be complex to interpret where a range of interventions co-exist.

6. Measurement What approaches are taken to measure health system performance in the study countries?

7. What is measured ? DataGeorgiaRwandaUgandaWest Bengal Demographic Census (2002) Census (2001) Health financing NHA (2004)NHA (2003) NHA (2001), public expenditure reviews, Tracking Study (2001)NHA (2001) Health outcomes incl. births and deaths RHS (2005) & MICS (2006), Vital registration & HMIS DHS (2005), HMIS (facility)DHS (2004)DHS (2005) Co-coverage of interventions MICS (2006) & RHS (2005)DHS (2005)DHS (2004)DHS (2005) Human resources HMISHR inventory HMIS Service provision HMIS & SAM (2005- 06) HMIS, SAM (2004) & Service Provision Assessment Survey (2001) HMIS, SAM (2004), Area Team assessmentsHMIS Quality n/a Various / accreditationn/a Vertical programme monitoring Immunisation, TB, HIV/AIDS Immunisation, malaria, HIV/AIDS, TB etc. Malaria, RCH, TB, Leprosy, Polio, HIV/AIDS etc. Disease surveillance IDRS Sentinel sites (HIV), early warning system, IDRS Sentinel sites (HIV)HMIS

8. Health financing How has information been used? Where are the gaps? What challenges remain?

9. Use of health financing information n Identification of financing gaps and advocacy for increased allocation of funds to health (Rwanda) n Health sector leadership and management of funds (Tanzania, Rwanda) n Equity of health financing in the health system (South Africa, Rwanda) n Protection against the financial burden of ill health (Mexico) n Resource allocation with the health sector (Rwanda)

10. Gaps in health financing information n Private health expenditures – difficult to collect compared to public and external health financing sources n Coverage of NHA relatively low in developing countries but expanding n Health financing data at decentralised levels for local decision- making n Financial burden of ill health and impact on impoverishment at the household level

11. National Health Accounts in Africa Number of NHA Rounds 1994 – 2004

12. Remaining challenges n Institutionalisation of NHA into the routine activities of Government n Underlying problems in Public Expenditure Management systems and data reliability n Timeliness of data (NHAs and household surveys) n Collection of private expenditure health financing data n Tension between disease expenditure and general health expenditure financial tracking n Addressing the needs of in-country policy makers vis-à-vis that of external agencies

13. Health care delivery How has information been used? Where are the gaps? What challenges remain?

14. Use of information: country examples West Bengal, India Aim: to monitor the performance of public sector programmes. Improve accountability and planning at national level n Standard service use indicators & regular meetings in PHC facilities Uganda Aim: to link health system performance monitoring to SWAPs and national policy process. Allows policy adjustment. n Data used in the annual health sector review process and to inform the development of annual plans n District league tables to rank performance of districts & motivate districts to improve indicators. n Tracking surveys – at the start of SWAP, 2001- to assess Govt systems (financial procedures, drug distribution, HR deployment)

15. Major gaps in measurement n Private sector – service use, service availability (infrastructure, human resources, services offered) n Vital events n Efficiency of health system n Quality of health care n Effective coverage

16. Remaining challenges n Low capacity and motivation to use data: –Locally –For decision-making or for policy initiatives n Lack of ownership by health providers, who are not involved in designing of monitoring procedure and indicators n Capacity for analysis concentrated at central level n Feedback to lower levels is limited, poor internal feedback n HMIS is often mistrusted n Selection of indicators often creates distortions n Information systems do not reflect move from project to system performance –India: critical milestones & vertical project indicators

17. Emerging issues

18. Data quality and reliability n Existing information systems, but data inaccessible or inappropriate to needs and policy process n Developing parallel monitoring frameworks rather than adapting & use of existing data: concerns for complexity and data reliability n HIS not always reflecting reform developments n Limited external data audit and reliance on single data sources (Rwanda, Uganda) n Technology involved in data collection, analysis and use often rely on bespoke software.

19. Parallel systems n Donor agenda regarding data collection, unsustainable n Data collection, analysis and use for policy is fragmented –Uganda/Nepal: lack of unified data linked to SWAPs –Private sector is often not covered (India/Uganda) n Multiple reporting requirements (Rwanda/India). n Lack of inter-sectoral information systems and unified quality standards. (Uganda/ Rwanda) n Vertical donors-supported programmes often function well in the short-term but may distort wider systems (e.g. Georgia & Angola)

20. Information flows & level of use n One-way traffic for information –Disaggregated data not available at sub-national level –Information intended to be used locally, is used at national level, or for different purpose reflecting governance & aid coordination n Information that is not aggregated nationally, less useful internationally n Governance and stewardship at local level needs to be able to draw effectively on aggregate & disaggregate data –Disaggregated data feeds effectively into local planning when linked to decentralised decision-making (TEHIP) n Peer comparisons at district level – productive vs unhelpful

21. Factors facilitating measurement & use of data n Health system monitoring embedded within reform process –SWAPs/ PRSP in Uganda, Rwanda; district autonomy (TEHIP) n Unintended consequences (Afghanistan) –Selective use of data internationally (user fees/HIV, in Uganda) n In post-conflict settings, the aid influx promotes monitoring health systems & early warning systems. Possible inefficiencies. n The importance of governance –Channels for policy exist (annual reviews, SWAPs meetings) & comparable timelines. –Communities and non-health system stakeholders involved n Large-scale data collection exercises are resource-intensive and not synchronised with the policy process (some In-DEPTH/ LSMS). n Technology, appropriate to context

22. Conclusions Effective health systems monitoring requires: n Capacity: to collect or use existing data, analyse, inform policy n Ownership n Coherence between domestic and external demands n Coherence between external agencies n Coherence between system-wide monitoring and vertical programmes performance measurement n Coherence between assessing the performance of different system elements n Domestic governance n Impact measurement to ensure sustainability/reform (scaling up) n Foster partnership between stakeholders

23. Acknowledgements GeorgiaGeorge Gotsadze IndiaBarun Kanjilal Rwanda Vianney Nizeyimana TanzaniaGraham Reid UgandaValeria Oliveira-Cruz Freddy Ssengoba