Presentation on theme: "The National Evaluation Platform"— Presentation transcript:
1 The National Evaluation Platform ApproachRobert E Black MD, MPHInstitute for International ProgramsBloomberg School of Public HealthJohns Hopkins University
2 Outline Why a new approach is needed National Evaluation Platforms (NEPs): The basicsCountry example: MalawiPracticalities and costs
3 Most current evaluations of large-scale programs aim to use designs like this No programImpactCoverageNo impactNo coverageTwo basic designs at present:Before-and-after evaluation in program areaComparison of program and non-program areas over time
4 But reality is much more complex General socioeconomic and other contextual factorsImpactCoverageRoutine health servicesInterventions in other sectorsOther healthprogramsProgramOther nutrition andhealthprograms
5 Mozambique How to evaluate the impact of USAID-supported programs? Traditional approach: intervention versus comparison areasSource:Hilde De Graeve,Bert Schreuder.
6 Mozambique Simultaneous implementation of multiple programs Separate, uncoordinated, inefficient evaluationsif anyInability to compare different programs due to differences in methodological approaches and indicatorsSource:Hilde De Graeve,Bert Schreuder.
7 New evaluation designs are needed Lancet, 2007Large-scale programsEvaluators do not control timetable or strength of implementationMultiple simultaneous programs with overlapping interventions and aimsContextual factors that cannot be anticipatedNeed for country capacity and local evidence to guide programmingBulletin of WHO, 2009Sources: Victora CG, Bryce JB, Black RE. Learning from new initiatives in maternal and child health. Lancet 2007; 370 (9593):Victora CG, Black RE, Bryce J. Evaluating child survival programs. Bull World Health Organ 2009; 87: 83.
8 National Evaluation Platforms: The Basics Lancet, 2011National Evaluation Platforms: The Basics
9 Builds on a common evaluation framework, adapted at country level Common principles (with IHP+, Countdown, etc.)Standard indicatorsBroad acceptance
10 Evaluation databases with districts as the units District-level databases covering the entire countryContaining standard information on:Inputs (partners, programs, budget allocations, infrastructure)Processes/outputs (DHMT plans, ongoing training, supervision, campaigns, community participation, financing schemes such as conditional cash transfers)Outcomes (availability of commodities, quality of care measures, human resources, coverage)Impact (mortality, nutritional status)Contextual factors (demographics, poverty, migration)Permits national-level evaluationsof multiple simultaneous programs
11 A single data base with districts as the rows Core Data Pointsfrom Health SectorCore Data Pointsfrom Other SectorsNutrition Surveillance SystemWomen’s educationRainfall patternsNational Stocks data baseDHSQuality Checking & Feedback to SourceDistrict…ChitipaKaronga….HMIS
12 Types of comparisons supported by the platform approach Areas with or without a given programTraditional before-and-after analysis with a comparison groupDose response analysesRegression analyses of outcome variables according to dose of implementationStepped wedge analysesIn case program is implemented sequentially
13 Evaluation platform Interim (formative) data analyses Are programs being deployed where need is greatest?Correlate baseline characteristics (mortality, coverage, SES, health systems strength, etc) with implementation strengthAllows assessment of placement biasIs implementation strong enough to have an impact?Document implementation strength and run simulations for likely impact (e.g., LiST)How to best increase coverage?Correlate implementation strength/approaches with achieved coverage (measured in midline surveys)How can programs be improved?Disseminate preliminary findings with feedback to government and partners(All analyses at district level)
14 Evaluation platform Summative data analyses Did programs increase coverage?Comparison of areas with and without each program over timeDose-response time-series analyses correlating strength of program implementation to achieved coverageWas coverage associated with impact?Dose-response time-series analyses of coverage and impact indicatorsSimulation models (e.g. LiST) to corroborate resultsDid programs have an impact on mortality and nutritional status?Dose-response time-series analyses correlating strength of program implementation with impact measures
15 The platform approach can contribute to all types of designs Having baseline information on all districts allows researchers to measure and control placement biasIn real life one cannot predict which districts will have strong implementation and which ones will notIn intervention/comparison designs, it is important to document that comparison districts are free of the interventionCollecting information on several outcomes allows assessment of side-effects of the program on other health indicators
17 Simultaneous implementation of multiple programs Separate, uncoordinated, inefficient evaluations (if any)Inability to compare different programs due to differences in methodological approaches and indicatorsThis shows the presence of the various funding partners across the various provinces. As you can see the howl coutnry is covered with their flags and logo´s.Technical agencies such as WHO and UNICEF are practically present in all provinces.Other mainly bilteral are only present in one (Denmark, EC, Finland, Flanders France) or two provinces (Catalunya, Ireland, Italy, Spain, Swiss, UK).Within each province an agency often covers only one or a few districts.
18 Malawi CCM scale-up limits use of intervention-comparison design … and implemented in Hard-to-Reach Areas! (March 2011)CCM supported in all districts beginning in 2009…Proportion of Hard-to-Reach Areas with ≥1 Functional Village Clinic, March 2011
19 Malawi adaptation of National Evaluation Platform approach National Evaluation Platform design using dose-response analysis, withDose = Program implementation strengthResponse = Increases in coverage;decreases in mortalityEvaluation Question:Are increases in coverage and reductions in mortality greater in districts with stronger MNCH program implementation?
20 Platform design overview Design elementData sources (sample = 29 districts)Documentation of program implementation and contextual factorsFull documentation every 6 months through systematic engagement of DHMTsQuality of care survey at 1st- level health facilitiesExisting 2009 data to be used for 18 districts; repeat survey in 2011Quality of care at community level (HSAs)Desirable to conduct in all 28 districts (Not included in this budget proposal)Intervention coverageDHS 2010, with samples of 1,000 households representative at district level in all 28 districtsDHS/MICS 2014 with samples representative at district level in all 28 districtsCostsCosting exercises in ≈ 1/3 of districts distributed by region and chosen systematically to reflect differences in implementation strategy or health system contextImpact (under-five mortality and nutritional status)End-line household survey (MICS or DHS?) in 2014Modeled estimates of impact based on measured changes in coverage using LiST
21 National Evaluation Platform: Progress in Malawi - 1 Continued district level documentation in 16 districtsPilot of cellphone interviews for community-level documentationStakeholder meeting in April 2011Full endorsement by the MOHPartners urged to coordinate around developing a common approach for assessment of CCM and non-CCM program implementation strengthNeed to allow sufficient implementation time to increase likelihood of impactMOH addressed letter to donors requesting support for platformPartners’ meetings in September and December 2011 to agree on plans for measuring implementation strength
22 National Evaluation Platform: Progress in Malawi - 2 All partners (SCF, PSI, WHO, UNICEF) actively monitoring CCM implementation in their districtsFunding secured for 16 of 28 districts; additional funding for remaining districts seems probableDiscussions under way about broadening platform to cover nutrition programsOther countries expressing interest! Mozambique, Bangladesh, Burkina Faso, …
23 Analysis Plan “Dose” “Response” CCM implementation strength (per 1,000 pop):CHWsCHWs trained in CCMCHWs supervisedCHWs with essential commodities availableFinancial inputsChange in Tx rates for childhood illnessesChange in U5M
24 Contextual Factors Categories Indicators Rainfall patterns ENVIRONMENTAL, DEMOGRAPHIC AND SOCIOECONOMICRainfall patternsAverage annual rainfall; seasonal rain patternsAltitudeHeight above sea levelEpidemicsQualitativeHumanitarian crisesSocio-economic factorsWomen’s education & literacy; household assets; ethnicity, religion and occupation of head of householdDemographicPopulation; population density; urbanization; total fertility rate; family sizeFuel costs!Added as this slowed program implementation inHEALTH SYSTEMS AND PROGRAMSUser feesChanges in user fees for IMCI drugsOther MNCH Health ProgramsThe presence of other programs or partners working in MNCH
25 Advantageous context for NEP strong network of MNCH partners implementing CCMadministrative structure decentralized to districtsSWAp II in development nowdistrict-level data bases (2006 MICS, 2010 DHS, Malawi Socio-Economic Database (MASEDA))DHS includes approx. 1,000 households in each district
27 Sample sizes must be calculated on a country-by-country basis Statistical power (likelihood of detecting an effect) will depend on:Number of districts in country (fixed; e.g. 28 in Malawi)How strongly the program is implemented, and by how much implementation affects coverage and mortalityHow much implementation varies from district to districtBaseline coverage levelsPresence of other programs throughout the districtsHow many households are included in surveys in each districtMay require oversampling
28 Practical arrangements Platform should be led by credible independent partner (e.g. University or Statistical Office)Supported by an external academic group if necessarySteering committee with MOH and other relevant government units (Finance, Planning), Statistical Office, international and bilateral organizations, NGOs, etc.
29 Main costs of the platform approach Building and maintaining database with secondary information already collected by othersRequires database manager and statistician/epidemiologist for supervisionMay require reanalysis of existing surveys, censuses, etcKeeping track of implementation of different programs at district levelRequires hiring local informants, training them and supervising their workAdding special assessments (costs, quality of care, etc)May require substantial investments in facility or CHW surveysOversampling household surveysMay require substantial investmentsBut this will not be required in all countries
30 Summary: Evaluation platform LimitationsObservational design (but no other alternative is possible)High cost particularly due to large size of surveysBut cheaper than standalone surveysRequires transparency and collaboration by multiple programs and agenciesAdvantagesAdapted to current reality of multiple simultaneous programs/interventionsIdentification of selection biasesPromotes country ownership and donor coordinationEvaluation as a continuous processFlexible design allows for changes in implementation