1. Steve Hodgins MCHIP/JSI (presenting) Marge Koblinsky MCHIP/JSI
Revisiting global benchmark indicators for maternal health Interventions for Impact in Essential Obstetric and Newborn Care Africa Regional Meeting, 21–25 February 2011
Steve Hodgins MCHIP/JSI (presenting)
Marge Koblinsky MCHIP/JSI

2. Why monitor? 2 For accountability
Whether it’s to our minister or to our donor, we have been entrusted with resources on the expectation that we will have something to show for how we’ve been making use of them. We need to demonstrate results. We report against agreed-upon indicators, to provide evidence that our work is accomplishing its intended objectives.
To direct our improvement efforts
As managers, whether at national level or within a health facility, we have a responsibility to bring about improvements in our programs and services. We want higher population coverage and better quality of care. This can only be accomplished by continuing to monitor; the results of our monitoring then tell us where we need to make adjustments in how we’re working. 2

3. What do we monitor? Overall program performance
Key sub-systems and processes
What do we monitor?
Both for those to whom we are accountable and for ourselves, we need to be able to track how well our programs are doing, overall.
That is to say, we need to be able to monitor overall program performance.
As program managers, we also need to be able to track the various key sub-systems and processes. 3

4. What makes a good indicator?
For these purposes of tracking overall program performance and what’s happening with regard to sub-systems and processes, what makes a good indicator?
It needs to reasonably closely approximate whatever the characteristic is that we’re particularly interested in
and
It needs to be feasible to collect on a frequent enough basis 4

5. What we measure is what we pay attention to.
Why is it important that we have the right set of indicators?
What we measure is
what we pay attention to.
Why is it important that we have the right set of indicators?
It we are held accountable for reporting on a set of indicators that is inappropriate or inadequate, this will tend to push us to make the wrong strategic choices
As a manager, if I am using an inappropriate or inadequate set of indicators I will not have the necessary information to effectively direct my efforts in improving performance of my program. 5

6. Global Benchmark Indicators
Note that from this point on, I will be focusing on the first of the two types of indicators I’ve been talking about—
those most commonly used as overall measures of program performance, what I will refer to as ‘benchmark indicators’
global benchmark indicators are used for a variety of purposes, for example:
for tracking progress on MDGs,
for planning purposes, for prioritization and projecting impact (LiST)
for holding countries and program managers accountable,
for reporting in global fora (e.g annual World Health Assemblies) and global publications (e.g. annual UNICEF reports),
as a basis for ‘results-based’ financing schemes
Let’s look at several that are used for child health….

7. Benchmark Indicators for Maternal Health
The primary indicator for MDG5 is MMR, which very closely approximates what we’re interested in influencing; unfortunately it is not generally feasible to measure this on a very frequent basis and even when we do measure it, there is considerable imprecision in the measured or modeled MMRs
The two secondary maternal health indicators for MDG5 are ANC and SBA delivery coverage;
both of them measure very important contacts between the system and beneficiaries, during the antenatal and intrapartum periods.
However, they measure only contact, i.e. an opportunity to provide a needed intervention or service
Is this adequate?

8. In this slide we see the maternal-newborn section of one of this year’s MDG Countdown country profiles – this happens to be for Ethiopia.
We have the secondary MDG5 indicators but several others as well…..
another contact measure – PNC
caesarian section rate, and
the proportion of pregnant women receiving intermittent preventive treatment for malaria

9. That’s all this indicator measures.
Clearly, what we’re really interested in, at the end of the day, is driving down the burden of maternal deaths. But, as you know better than I, generally we are unable to measure maternal mortality on a very frequent basis and even when we are able to measure it, we have important precision problems.
  So, for year-to-year monitoring of progress, we rely on other indicators. This and the next 2 slides show data on 3 of the indicators which take pride of place among global benchmark indicators for maternal health. All of these are among the MDG5 secondary indicators.
  In these slides, I am using the national MMR estimates modeled for 2008, from Hogan’s recent Lancet paper for the 30 countries with the greatest number of maternal deaths which, together, account for about 90% of all maternal deaths globally. Data on SBA delivery, ANC and CPR rates are from the most recent national surveys, generally DHS or MICS.
  All 30 of these countries have MMR’s above 200 except China, where MMR is only 40.
China makes it onto this list by virtue of its very large population. Eyeballing this distribution, there appears to be some correlation. Pearson r, calculated across the 30 countries, is -0.44, indicating a moderate negative correlation.
Recall that taking the square of Pearson’s r gives you the proportion of variance explained. So, including all 30 data points, SBA delivery rate statistically accounts for about 19% of the between-country variation in MMR.
  But recall also that Pearson r is not a very robust measure; it tends to be overly sensitive to outliers or extreme values. Restricting the sample by removing the extreme cases of China and Afghanistan, r drops to -0.21, a weak negative correlation, which means that differences in SBA delivery rates across this set of 28 countries account for only 4% of the between-country variation in MMR.
Beyond the degree of correlation, it is striking that at very similar SBA delivery rates, MMR can vary enormously. Compare, for example Bangladesh and Afghanistan or Burkina Faso and Malawi. Likewise, at similar levels of MMR, SBA delivery rates vary widely. But this shouldn’t come as a surprise. Recall what SBA delivery rates measure:
If from MoH HMIS data, in the numerator we have all deliveries documented as having been attended by a physician, midwife or other cadre classified by the MoH as an SBA; in the denominator, we have an estimate of total deliveries.
If based on survey data, we have the proportion of deliveries which, according to mothers surveyed, were attended by someone they thought belonged to one of the categories above.
That’s all this indicator measures.

10. In this slide, correlation is less evident.
Doing the math, ANC1 shows a weak negative correlation with MMR (r=-0.35), or 12% of the between-country variance in MMR explained by ANC. Excluding the extreme value, Afghanistan, R drops to -0.12, i.e. almost no correlation—1% of variance explained.
Recall again what is being measured. Like SBA delivery rate, the proportion of pregnant women receiving at least 1 ANC visit tells us that a contact occurred but it really tells us nothing about what was actually done during that contact.
So, we need to acknowledge that although these are important and appropriate indicators to track, they tell us little more than that we’ve been able to get these women through the front door into our clinics and hospitals. They tell us nothing about content and quality of care.

11. What’s now being done?
For those involved globally in maternal health program performance measurement, this problem of an inadequate set of benchmark indicators has been recognized for some time.
How is this being addressed?
A process has begun, led by WHO/ RHR and MPS and supported by MCHIP/USAID and CDC, revisiting the issue of benchmark indicators used for tracking maternal health program performance.
A group of maternal health measurement experts was convened in Delhi in September 2010 to table the issue; confirming that there was a shared perception that such work should be prioritized.
Following up on that meeting, a smaller group of key maternal health measurement specialists conducted a two day working meeting in Geneva in December. At that time several key areas were identified for further development and focal point persons were assigned to carry each of these forward over the coming several months, with a view to sharing results and producing new global guidance on benchmark indicators late this year. The areas identified for development included:
Oxytocin (or other suitable uterotonic) in the 3rd stage         
MgSO4 for severe pre-eclampsia and eclampsia
Quality of intrapartum care (measured through intrapartum mortality)
Obstetrical near-miss
Caesarian section (e.g. by absolute obstetrical indications; disaggregating by district, by socio-economic status) 11

12. How is this relevant to you?
What can we expect this coming year?
Not all of these areas are at the same stage of development
It is likely that by the fall, for several of them, there will be fairly definitive new recommendations for how Ministries of Health track MH program performance.
For others, the process will likely take somewhat longer, as new indicators and ways of collecting them through routine health information systems are tested and validated.
Over the coming year, in your country you may hear proposals about early tentative implementation of new performance monitoring indicators and approaches. In some countries, there may already be tracking of indicators closely related to these areas. For those involved in this effort globally, we are particularly interested in looking at settings where there has already been some experience in this area. 12

13. For further information, you can contact me at:
wwww.mchip.net
Follow us on: 13