Presentation on theme: "The National Evaluation Platform Approach Robert E Black MD, MPH Institute for International Programs Bloomberg School of Public Health Johns Hopkins University."— Presentation transcript:
The National Evaluation Platform Approach Robert E Black MD, MPH Institute for International Programs Bloomberg School of Public Health Johns Hopkins University
Outline 1.Why a new approach is needed 2.National Evaluation Platforms (NEPs): The basics 3.Country example: Malawi 4.Practicalities and costs
Most current evaluations of large-scale programs aim to use designs like this Impact Coverage Program No impact No coverage No program
But reality is much more complex General socioeconomic and other contextual factors Impact Coverage Routine health services Interventions in other sectors Other health programs Program Other nutrition and health programs
Mozambique How to evaluate the impact of USAID- supported programs? Traditional approach: intervention versus comparison areas Source: Hilde De Graeve, Bert Schreuder.
Mozambique Simultaneous implementation of multiple programs Separate, uncoordinated, inefficient evaluations – if any Inability to compare different programs due to differences in methodological approaches and indicators Source: Hilde De Graeve, Bert Schreuder.
New evaluation designs are needed Large-scale programs Evaluators do not control timetable or strength of implementation Multiple simultaneous programs with overlapping interventions and aims Contextual factors that cannot be anticipated Need for country capacity and local evidence to guide programming Lancet, 2007 Bulletin of WHO, 2009 Sources: Victora CG, Bryce JB, Black RE. Learning from new initiatives in maternal and child health. Lancet 2007; 370 (9593): Victora CG, Black RE, Bryce J. Evaluating child survival programs. Bull World Health Organ 2009; 87: 83.
NATIONAL EVALUATION PLATFORMS: THE BASICS Lancet, 2011
Builds on a common evaluation framework, adapted at country level Common principles (with IHP+, Countdown, etc.) Standard indicators Broad acceptance
Evaluation databases with districts as the units District-level databases covering the entire country Containing standard information on: Inputs (partners, programs, budget allocations, infrastructure) Processes/outputs (DHMT plans, ongoing training, supervision, campaigns, community participation, financing schemes such as conditional cash transfers) Outcomes (availability of commodities, quality of care measures, human resources, coverage) Impact (mortality, nutritional status) Contextual factors (demographics, poverty, migration) Permits national-level evaluations of multiple simultaneous programs Permits national-level evaluations of multiple simultaneous programs
A single data base with districts as the rows District…… Chitipa Karonga …. Core Data Points from Health Sector Core Data Points from Other Sectors HMIS DHS National Stocks data base Nutrition Surveillance System Rainfall patterns Women’s education Quality Checking & Feedback to Source
Types of comparisons supported by the platform approach Areas with or without a given program – Traditional before-and-after analysis with a comparison group Dose response analyses – Regression analyses of outcome variables according to dose of implementation Stepped wedge analyses – In case program is implemented sequentially
Evaluation platform Interim (formative) data analyses Are programs being deployed where need is greatest? –Correlate baseline characteristics (mortality, coverage, SES, health systems strength, etc) with implementation strength –Allows assessment of placement bias Is implementation strong enough to have an impact? –Document implementation strength and run simulations for likely impact (e.g., LiST) How to best increase coverage? –Correlate implementation strength/approaches with achieved coverage (measured in midline surveys) How can programs be improved? –Disseminate preliminary findings with feedback to government and partners (All analyses at district level)
Evaluation platform Summative data analyses Did programs increase coverage? –Comparison of areas with and without each program over time –Dose-response time-series analyses correlating strength of program implementation to achieved coverage Was coverage associated with impact? –Dose-response time-series analyses of coverage and impact indicators –Simulation models (e.g. LiST) to corroborate results Did programs have an impact on mortality and nutritional status? –Comparison of areas with and without each program over time –Dose-response time-series analyses correlating strength of program implementation with impact measures
The platform approach can contribute to all types of designs Having baseline information on all districts allows researchers to measure and control placement bias In real life one cannot predict which districts will have strong implementation and which ones will not In intervention/comparison designs, it is important to document that comparison districts are free of the intervention Collecting information on several outcomes allows assessment of side-effects of the program on other health indicators
COUNTRY EXAMPLE: CCM IN MALAWI
Simultaneous implementation of multiple programs Separate, uncoordinated, inefficient evaluations (if any) Inability to compare different programs due to differences in methodological approaches and indicators
Malawi CCM scale-up limits use of intervention-comparison design CCM supported in all districts beginning in 2009… … and implemented in Hard-to-Reach Areas! (March 2011) Proportion of Hard-to-Reach Areas with ≥1 Functional Village Clinic, March 2011
Malawi adaptation of National Evaluation Platform approach National Evaluation Platform design using dose- response analysis, with D OSE = P ROGRAM IMPLEMENTATION STRENGTH R ESPONSE = I NCREASES IN COVERAGE ; DECREASES IN MORTALITY Evaluation Question: Are increases in coverage and reductions in mortality greater in districts with stronger MNCH program implementation?
Platform design overview Design elementData sources (sample = 29 districts) Documentation of program implementation and contextual factors Full documentation every 6 months through systematic engagement of DHMTs Quality of care survey at 1 st - level health facilities Existing 2009 data to be used for 18 districts; repeat survey in 2011 Quality of care at community level (HSAs) Desirable to conduct in all 28 districts (Not included in this budget proposal) Intervention coverage DHS 2010, with samples of 1,000 households representative at district level in all 28 districts DHS/MICS 2014 with samples representative at district level in all 28 districts Costs Costing exercises in ≈ 1/3 of districts distributed by region and chosen systematically to reflect differences in implementation strategy or health system context Impact (under-five mortality and nutritional status) End-line household survey (MICS or DHS?) in 2014 Modeled estimates of impact based on measured changes in coverage using LiST
National Evaluation Platform: Progress in Malawi - 1 Continued district level documentation in 16 districts Pilot of cellphone interviews for community-level documentation Stakeholder meeting in April 2011 −Full endorsement by the MOH −Partners urged to coordinate around developing a common approach for assessment of CCM and non-CCM program implementation strength −Need to allow sufficient implementation time to increase likelihood of impact −MOH addressed letter to donors requesting support for platform Partners’ meetings in September and December 2011 to agree on plans for measuring implementation strength
National Evaluation Platform: Progress in Malawi - 2 All partners (SCF, PSI, WHO, UNICEF) actively monitoring CCM implementation in their districts Funding secured for 16 of 28 districts; additional funding for remaining districts seems probable Discussions under way about broadening platform to cover nutrition programs Other countries expressing interest! Mozambique, Bangladesh, Burkina Faso, …
Analysis Plan “Dose” CCM implementation strength (per 1,000 pop): CHWs CHWs trained in CCM CHWs supervised CHWs with essential commodities available Financial inputs “Response” Change in Tx rates for childhood illnesses Change in U5M
Contextual Factors CategoriesIndicators ENVIRONMENTAL, DEMOGRAPHIC AND SOCIOECONOMIC Rainfall patternsAverage annual rainfall; seasonal rain patterns AltitudeHeight above sea level EpidemicsQualitative Humanitarian crisesQualitative Socio-economic factors Women’s education & literacy; household assets; ethnicity, religion and occupation of head of household Demographic Population; population density; urbanization; total fertility rate; family size Fuel costs!Added as this slowed program implementation in HEALTH SYSTEMS AND PROGRAMS User feesChanges in user fees for IMCI drugs Other MNCH Health Programs The presence of other programs or partners working in MNCH
Advantageous context for NEP strong network of MNCH partners implementing CCM administrative structure decentralized to districts SWAp II in development now district-level data bases (2006 MICS, 2010 DHS, Malawi Socio-Economic Database (MASEDA)) DHS includes approx. 1,000 households in each district
PRACTICALITIES AND LIMITATIONS
Sample sizes must be calculated on a country-by-country basis Statistical power (likelihood of detecting an effect) will depend on: – Number of districts in country (fixed; e.g. 28 in Malawi) – How strongly the program is implemented, and by how much implementation affects coverage and mortality – How much implementation varies from district to district – Baseline coverage levels – Presence of other programs throughout the districts – How many households are included in surveys in each district May require oversampling
Practical arrangements Platform should be led by credible independent partner (e.g. University or Statistical Office) Supported by an external academic group if necessary Steering committee with MOH and other relevant government units (Finance, Planning), Statistical Office, international and bilateral organizations, NGOs, etc.
Main costs of the platform approach Building and maintaining database with secondary information already collected by others Requires database manager and statistician/epidemiologist for supervision May require reanalysis of existing surveys, censuses, etc Keeping track of implementation of different programs at district level Requires hiring local informants, training them and supervising their work Adding special assessments (costs, quality of care, etc) May require substantial investments in facility or CHW surveys Oversampling household surveys May require substantial investments But this will not be required in all countries
Summary: Evaluation platform Advantages – Adapted to current reality of multiple simultaneous programs/interventions – Identification of selection biases – Promotes country ownership and donor coordination – Evaluation as a continuous process – Flexible design allows for changes in implementation Limitations – Observational design (but no other alternative is possible) – High cost particularly due to large size of surveys But cheaper than standalone surveys – Requires transparency and collaboration by multiple programs and agencies