1. “Concepts in evaluation of vaccination programs in low income countries" Jagrati Jani Section for International Health Department of Community Medicine Institute for Health and Society Faculty of Medicine University of Oslo

2. Presentation Outline Introduction Epidemiological transition Evaluation conceptual framework Expanded Program on Immunisation Goals and main Indicators Vaccine Coverage and health Information System Disease surveillance and health Information System Final remarks

3. Transition in the population Demographic Epidemiological Nutrition transition transition transition High fertility/mortality High incidence High prevalence of communicable diseases undenutrition Socioeconomic Vaccination Safe water Improved diet Decreased fert./mort Reduction in infectious diseases Reduced undernutrition Increased overnutrition Rise of Chronic Diseases Popkin et al

4. Evaluation Is a systematic investigation of the structure, activities and results of public health programs –Achieve its goals and objectives Source: Center for Disease Control (CDC) Atlanta 2009

5. Evaluation methods StartAIM Planning an evaluation Method Qualitative, Quantitative or Both “HARD AND OBJECTIVE, SOFT AND SUBJECTIVE” Cross sectional Retrospective Prospective Data collection instrument Primary data (use of research instruments) Questionnaire Interviews Observations Secondary data (data already existing) Statistics on health Reports Field work How to apply the findings Dissemination Data collection Data analyze Interpretation of findings Report $

6. Inputs Data Finances Sustainability Activities Outputs Plans Number of vaccines administrated Number train staff Health system Service availability Quality of services Outcomes Targeted population receiving the intervention Vaccine coverage Vaccine Effectiveness Impact Disease incidence Mortality, morbidity Effectiveness Efficiency Impact Aim of the program Impact evaluation Formative evaluation Process evaluation Conceptual framework Adapted from the source: Health Care Evaluation. Open University Press, 2005

7. EPI goal To reduce mortality and morbidity from the targeted diseases among the target group. Indicators Vaccine coverage and vaccine efficacy Incidence of the disease

8. Output: Service availability Health system Health facility Curative care Preventive care vaccination Outreach program Supplementary vacc. program Health facility

9. Cold Chain Vaccine admin And safety Knowledge Immunization Program Contact with the client Community level Availability Inventory stocks Condition and maintenance of the equipment Vaccine & transport management information management & register Data collection Data register

10. Factors influencing the programme performance: Intrinsic factors Change in resources -External donors -National commitment Programme structure, management or activities -Changes in vaccination schedule -Vaccine shortage -Additional activities- campaigns -Human resources Extrinsic factors Political stability Population accuracy Accessibility to the HF Acceptability ( BCG Magude)

11. The reasons not to vaccinate Bivariate analysisBinary logistic regression analyses Variable Completely vaccinated N=479 (%) Incomplet. vaccinated N=189 (%) OR (95% CI) P value OR* (95% CI) P value Administrative posts Magude village Out of Magude village 392 (81.8) 85 (17.7) 104 (55.0) 84 (44.4) 1 3.72 ( 2.53-5.49) <0.00 1 1 2.89 (1.80 -4.64) <0.001 Time to arrive to HF & N= 348 N=281 53.6 (SD 41) N=95 84.64 (SD 55.6) <0.00 1 1 3.64 ( 1.71- 7.74) 0.001 Mother’s schooling Yes No 264 (55.1) 207 (43.2) 51 (26.9) 135 (71.4) 1 3.38 (2.30 -4.97 ) <0.00 1 1 2.24 (1.41-3.56) 0.001 EPI information Yes No 123 (25.6) 352 (73.4) 19 (10.0) 169 (89.4) 1 3.11 ( 1.81- 5.40) <0.00 1 2.02 (1.19 -3.42)0.009 Child born Inside of Mozambique outside of Mozambique 465 (97.0) 10 (2.0) 172 (91.0) 16 (8.4) 1 4.33 (1.81- 10.47) <0.00 1 1 5.20 (3.35- 11.51) <0.001 Place of delivery HF delivery Home delivery 384 (80.1) 88 (18.3) 123 (65.0) 64 (33.8) 1 2.27 (1.52- 3.38) <0.00 1 1 1.78 (1.28 – 3.36) 0.03 Source: J. V. Jani, C. De Schacht, I.V. Jani and G. Bjune: Risk factors for incomplete vaccination and missed opportunity for immunization in rural Mozambique. BMC Public Health, 2007

12. The Cow-Pock. The Cow-Pock or the Wonderful Effects of the New Inoculation! by James Gillray was published in England in 1802 by the Anti-Vaccine Society. The etching, which shows Edward Jenner among patients in the Small Pox and Inoculation Hospital at St Pancras (London), suggests the transformation into cows of individuals vaccinated by Jenner Vaccine safety–vaccine benefits: science and the public’s perception. Nature Review Immunology, Nov 2001

13. Output Challenges Vaccination Immunization Cold chain Vaccination technique Vaccination at right age ( Mab interference ) Vaccine effectiveness Outcome Impact

14. Inputs Activities Outputs Outcomes Targeted population receiving the intervention Vaccine Coverage Vaccine Effectiveness Impact Effectiveness Sustainability Efficiency Impact Aim of the program Process evaluation Conceptual framework Adapted from the source: Health Care Evaluation. Open University Press, 2005

15. Vaccination Immunization Accessibility (hard-to-reach pop) Health services ( coverage, staff, availability of vaccines etc.- MOV) Unstable vaccine supplies (stock out, release of funds)- MOV Demand/utilisation still low due to lack of awareness or illegal payments or side Effects Weak register management Outcome Challenges Vaccine coverage Outcome Impact

16. Challenges outcome Vaccine coverage Vaccine effectiveness Disease control Herd Immunity Disease elimination Vaccination Immunization Cold chain Vaccination technique Vaccination at right age ( Mab interference ) Health status of the individual Vaccine failure (hard-to-reach population) Genetic determination of MHC Major Histocompatibility Complex (MHC) Human Leukocyte Ag Demand Laboratory surveillance

17. An example of Vaccine Coverage Sum (n* of children vaccinated) Vaccine coverage =____________________________ Sum (n* of children in target group) Mathematician (exact number) Clinician EPI technician (work performance indicator) Mother Evaluator (hot issue)

18. Methods of assessing immunisation coverage Administrative method Sum ( n* of children vaccinated) Vaccine coverage % = __________________________ Sum ( n* of children in target group) Integral part of service delivery Data useable at all levels Prompt analyses and action is possible Survey method Sum ( n* of children vaccinated) Vaccine coverage % = ________________________ Sum ( n* of children surveyed) Is a complementary method and addresses denominator Issues Recall biases RH card 30 by 7 cluster sample N= 210

19. Numerator issues Vaccination after target age included? Campaigns included? ( polio-measles plus Vit A) Private sector included? Vaccine coverage

20. How good is the denominator? –Quality of census data/ Vital registers –Method of population projection ( 2,6%) –Uncertainty at the local level –Migration –Major population shifts ( refugees) –Births or surviving infants

22. Coverage more 100% Problems with denominator Problems with numerator ( the case of Magude District) Strategies to reach high coverage All opportunities to immunised National vaccination days

23. Fragmentation of different data sources Finance IS HR IS Routine information system Vital Registration Demographic Health Survey HH survey Other HH Surveys PopulationC ensus Disease Surveillance Health System Vaccine ISOther PHC programs IS

24. An example of DTP

25. DTP coverage trend Figs: Survey-based coverage of DTP3 from bidirectional distance dependent regression with 95% uncertainty intervals from 1986-1996 by Global burden of disease study 20001990 Source: The Lancet 2008; 372: 2031-72

26. Additional children immunized and immunization service support payments in countries receiving GAVI support up to 2006 Source: The Lancet 2008; 372: 2031-72

27. Challenges to explain vaccine coverage discrepancies Intentionally over report or Errors in stages of the data gathering or reporting process.

28. Challenges to evaluate outcome ( VE) Surveillance system Challenges Information gap on new diseases Depends on the coverage of the health system

29. Flow of the disease surveillance in the health system, Mozambique Source: BMC Infectious Diseases 2006, 6:29

32. Source: BMC Infectious Disease, 2006; (6)-1-9

35. Final remark Vaccine coverage is not the only indicator (immunity, equity, efficiency and sustainability all matter) December 2008 was 60th anniversary of the Universal Declaration of Human Rights – right to health.

36. Vaccine Coverage Challenges There are problems with data quality and unharmonized donor requirements for M&E—a situation of too much data, not enough information Indicators that need more than one data base

37. Unprecedented commitment by the world’s leaders to tackle the most basic forms of injustice and inequality in our world: poverty, illiteracy and ill-health. Globalization of humanities basic values: human rights and human development including human health of all The MDGs are drawn from the actions and targets contained in the Millennium Declaration that was adopted by 189 nations-and signed by 147 heads of state and governments during the UN Millennium Summit in September 2000UN Millennium Summit The eight MDGs break down into 21 quantifiable targets that are measured by 60 indicators Millennium Development Goals

38. Knowledge Cold Chain Vaccine supply Standardized packages of information Disease surveillance system