1. Donna Rickert, M.A., Dr.P.H. Abigail Shefer, M.D. National Immunization Program Immunization Services Division Health Services Research and Evaluation Branch April, 2002 A Model for Immunization Assessment and Referral in Non-Medical Settings Counting the Shots:

2. “The task of the public health agency has been not only to define objectives for the health care system … but also to find means to implement health care goals within a social structure.” Institute of Medicine, The Future of Public Health

3. Over the past decade, major changes in our national health and welfare delivery systems have presented special challenges for the WIC-Immunization initiative:  State Medicaid agencies have increasingly relied on private managed care organizations to provide services to indigent children  The service delivery role of local public health agencies has been weakened  WIC’s direct link to the health care system has been weakened Special Challenges …

4. As fewer public health departments provide services to women and children  public health officials are turning to WIC to help address the needs of low income children  WIC has become the virtual gateway to health services for low-income preschool children such as those related to overweight and obesity, anemia, HIV, and blood lead level screening in addition to immunization Special Challenges …

5. In addition, WIC Programs  Are seeing significant health and demographic changes in the low-income populations that it serves  Are facing difficulties in recruiting and keeping skilled staff  Are having difficulties in funding information technology to manage program operations while simultaneously enhancing service delivery Special Challenges …

6.  Addressed socioeconomic disparities in pediatric immunization coverage levels  Directed Secretaries of DHHS and Agriculture to work together to  make IZ screening and referral a standard part of WIC certification  develop a national strategic plan to improve IZ coverage rates in WIC children  report back on progress The White House Memorandum, December, 2000

7.  Sets minimum guidelines for IZ screening and referral  Requires documented immunization history  Applies to infants and children under age 2 years  Screening and referral to be done at certification visits  WIC clinic staff will count DTaP immunizations only   1 dose of DTaP by age 3 months   2 doses of DTaP by age 5 months   3 doses of DTaP by age 7 months   4 doses of DTaP by age 19 months The Final WIC Policy Memorandum, August, 2001

8. Diagram of the process described in the WIC policy memorandum Child comes to WIC clinic certification visit? Classify child as 4:3:1:3 UTD Please bring next time DTP UTD for age? Refer to provider yes no yes no No IZ assessment required documented IZ record? yes no

9. In studies using NIS data, the provider’s portion of the record is considered the more accurate index of the child’s true immunization status. The household portion of the record is assumed to be a less accurate representation. In most WIC clinic settings, only the household-based record will be available. For this reason, our aim was to see how closely the household- based record approximates the provider-based record. Rationale

10. Primary objective:  To estimate the percentage of WIC children who, by the new WIC minimum assessment criteria, will be correctly classified as either up-to-date (UTD) or not for the universal 4:3:1:3 pediatric vaccination series  To see how this compares with the percentage that would be correctly classified if all 4 antigens were assessed Purpose of this analysis …

11. We used approximately 6,000 household and provider records from the 2000 National Immunization Survey (NIS) to calculate the sensitivity, specificity, and test efficiency of using DtaP shots as a predictor of UTD status for the universal 4:3:1:3 pediatric immunization series. Method Data Source and Statistics

12. Sample size and inclusion criteria: Method … Total NIS 2000 Survey Records = 34,087 Ever enrolled in WIC = 17,451 (51% of NIS 2000) Shot card used for household reporting = 8,617 (25% of NIS 2000) Adequate provider data for verification = 6,277 (18% of NIS 2000)

13. 4 doses of DTaP or DTP at 2, 4, 6, and 15 - 18 months 3 doses of Polio at 2, 4, and 6 - 18 months 1 dose of MMR or MCV at 12 - 15 months 3 doses of Hib at 2, 4, 6, and 12 - 15 months Method: Definitions Up-to-Date by the 4:3:1:3 ACIP schedule is defined in terms of the age-specific vaccination schedule for the following 4 vaccines:

14. NIS Household Survey  child’s age at each vaccination NIS Provider survey  child’s UTD status for each vaccine at 3, 5, 7, 13, 19, and 24 months We assigned bivariate UTD status variables  Household DTaP  Household 4:3:1:3  Provider 4:3:1:3 Method: Variable Specification

15. Let “test” refer to any measure used to make a decision about the “true status” when information about the true status is incomplete.  test 1: the household-reported DTaP count  test 2: the household-reported 4:3:1:3 count We compared test 1 and test 2 to see if they differed in their ability to predict the child’s true 4:3:1:3 UTD true status as shown in the provider record. Method: Definitions

16. Sensitivity of a test measures the percentage of truly underimmunized children who are correctly identified as such by the test. Specificity of a test measures the percentage of truly UTD children who are correctly identified as such by the test. Test efficiency measures the total percentage of times the test gives the correct answer, relative to all times the test is given. Method: Definitions

17. SAS To develop the enhanced database SUDAAN To conduct the weighted crosstabulation procedures Excel To calculate sensitivity, specificity, and test efficiency statistics Method: Statistical Analyses

18. Research Questions … What is the estimated nationwide difference in immunization coverage rates between WIC vs non-WIC children under age 2? Socioeconomic Disparities

19. Differences in Immunization Completion Rates, WIC vs Non-WIC Children, 2000

20. Research Question: How good is the household-based record of UTD status for DTaP at identifying children who, by the provider-based record, are underimmunized? How does this compare with the household-based count of all 4 antigens? Comparative Sensitivity

21. Results: Comparative Sensitivity

22. How good is the household-based record of UTD status for DTaP at identifying children who, by the provider-based record, are truly UTD for the complete 4:3:1:3 series? How does this compare with the household-based count of all 4 antigens? Research Question: Comparative Specificity

23. Results: Comparative Specificity

24. How often does the Household DTaP count accurately reflect the provider- based overall 4:3:1:3 UTD status? How often does the Household count of all 4 antigens accurately reflect the provider- based 4:3:1:3 UTD status? Research Question … Comparative Test Efficiency

26. Summary The NIS 2000 data confirm that children under age 2 who have ever been enrolled in WIC are significantly more likely to be underimmunized than those who have never been enrolled. The overall disparity is 7%.

27. Summary The household DTaP count and the household 4:3:1:3 count are both imperfect predictors of true 4:3:1:3 UTD status.  If the DTaP count alone is used, 70% of underimmunized children, on average, will be identified.  If all 4 antigens documented in the household record are counted, 77% of underimmunized children, on average, will be identified  Therefore the count of all 4 antigens is a more sensitive predictor in that it is slightly better (7%) at identifying truly underimmunized children. However…

28. Summary  If the DTaP count is used, 86% of truly UTD children, on average, will be identified as such.  If all 4 antigens from the household record are counted, 82% of truly UTD children, on average will be identified as such.  Therefore the DTaP count is a more specific predictor of true UTD status in that it is slightly better (4%) at identifying truly UTD children. Moreover,  Overall test efficiency is slightly higher (1% on average) for the DTaP count, making it a more efficient predictor of true immunization status.

29. Conclusions  Training non-medical staff to assess completion status for 1 vaccination is simpler than training them to assess it for 4 or 5  The immunization assessment process should go more quickly and is therefore more efficient in terms the demands on staff time  The likelihood that non-medical staff will make an error is smaller when only one vaccine is assessed. Use of the DTaP count as a predictor of true 4:3:1:3 UTD Status has specific advantages:

30. Conclusions  The higher specificity of the DTaP count means that children are less likely to be inappropriately referred to an immunization provider  If the gains in simplicity and efficiency of assessment result in more clinics being able/willing to evaluate children’s immunization status, this may offset the loss of sensitivity, and … Advantages … (continued)

31. Conclusions  Success in this initiative will likely strengthen the partnership, which will  Set the stage for assisting WIC in developing achievable objectives toward improving immunization coverage in WIC children, as well as  Serve as a model for developing immunization linking initiatives with other federal/state programs, such as Housing and Urban Development, Medicaid and TANF Advantages … (continued)

32. Contact Information: e-mail: djr7@cdc.gov phone: (404) 639-8827