1. Quality Assessment of Minnesota Newborn Hearing Screening Data: A Pilot Study National EHDI Conference March 26, 2007 Salt Lake City Penny Hatcher, Supervisor and Grant Director Yaoli Li, CDC EHDI Coordinator Nicole Brown, HRSA UNHSI Coordinator Katie James, UNHSI Student Worker Sarah Solarz, EHDI Student Worker Judy Punyko, MDH Epidemiologist Minnesota Department of Health (MDH) Community & Family Health Newborn & Child Screening Unit

2. 2 Faculty Disclosure Information In the past 12 months, I have not had a significant financial interest or other relationship with the manufacturer(s)of the product(s) or provider(s) of the service(s) that will be discussed in my presentation. This presentation will not include discussion of pharmaceutical or devices that have not been approved by the FDA or if you will be discussing unapproved or off- label uses of pharmaceuticals or devices.

3. 3 Study Purpose In compliance with the CDC guidelines for evaluating public health surveillance systems… Assess the quality of Minnesota newborn hearing screening data -Validity -Reliability

4. 4 Methodology – Planning Phase Develop partnership with –Vital Records (CHS) –Birth Defects Information System (EH) –Newborn Bloodspot Screening (PHL) –Newborn Hearing Screening Data fields Medical record abstraction form

5. 5 Medical Record Abstraction Form (Infants)

6. 6 Medical record abstraction form (7) BIRTH DATE From BC: 01/10/2005 mm dd yyyy *9-fill fields if missing (11) MULTIPLE BIRTHS / BIRTH ORDER (a, b, etc.) 1 = Single Birth 2 = Twin ___ (a or b) 3 = Triplet ___ (a, b, or c) 4 = Quadruplet ___ (a, b, c, or d) 5 = Quintuplet ___ (a, b, c, d, or e) 6 = Other 9 = Unknown, not stated, unclassifiable BC DATA 2 a (15) LEFT EAR SCREEN RESULTS, 1 MONTH 1 = Pass 2 = Fail 9 = Not screened/missing BS DATA 1

7. 7 Methodology – Planning Phase Select 20 MN hospitals with 15 births in 2005 –Hospitals rank-ordered by size (i.e. # births) –Every 5 th hospital chosen e.g. Hospital Size A100 start B93 C90 D86 E82 F79 G70

8. 8 Methodology – Implementation Two graduate student workers oriented by BDIS staff Vital Records randomly selects mothers (n = 200) and their infants (n = 200) from birth certificates (Total N = 400)

9. 9 Methodology – Implementation Letters sent to hospitals Phone calls made to confirm appointments List of 10 infant and 10 mother records faxed to each hospital

10. 10 Students hit the road…

11. 11 Methodology – Data Collection At each hospital… –Collect and record information from medical records Upon return to MDH… –Students double enter data

12. 12 Methodology – Data Analysis Assess inter-rater reliability –How well do students data agree? –If discrepancies, determine which answer is correct –Create final (corrected) database

13. 13 Methodology – Data Analysis Merge medical record (MR) data with hearing screening (HS) data

14. 14 Methodology – Data Analysis Clean the merged dataset –e.g. Duplicate records Use only records with lab or loose (i.e. from birth hospital) designation as data source. NAME BC NUMBER DOB SOURCE Baby, Girl 2005-MN-0123456 1/1/2005 lab Baby, Girl 2005-MN-0123456 1/1/2005 C1 refer Baby, Girl 2005-MN-0123456 1/1/2005 refer

15. 15 Methodology – Data Analysis Clean the merged dataset –e.g. Missing records 5 infants –2 died (no hearing screen done) –1 transferred to NICU (no record of HS at birth hospital) –2 with evidence of HS in medical record but were not in HS database

16. 16 Methodology – Data Analysis Missing values –Recoded into a no/unknown category Weighting scheme

17. 17 Methodology – Data Analysis Analysis of categorical (yes/no) variables included calculations of: –Sensitivity –Specificity –Positive predictive value

18. 18 Sensitivity Medical Record – Left Ear Results HS data – Left Ear Results Yes Pass No Pass Total Yes Pass16013173 No Pass72027 Total16733200 Sensitivity = 160 / 167 = 95.8%

19. 19 Specificity Medical Record – Left Ear Results HS data – Left Ear Results Yes Pass No Pass Total Yes Pass16013173 No Pass72027 Total16733200 Specificity = 20 / 33 = 60.6%

20. 20 Positive Predictive Value Medical Record – Left Ear Results HS data – Left Ear Results Yes Pass No Pass Total Yes Pass16013173 No Pass72027 Total16733200 PVP = 160/173 = 92.5

21. 21 Results – Left Ear/Right Ear Screens Characteristics HS MR % Missing In HS MR Sensitivity (%) (SE) Specificity (%) (SE) PVP (%) (SE) % left ear pass 98.3 97.112.0 14.099.0 (0.6) 48.0 (15.2) 94.6 (2.2) % right ear pass 96.6 97.712.0 14.098.0 (1.1) 52.5 (17.4) 95.4 (2.0)

22. 22 Results – Left Ear/Right Ear Screens % of infants with pass results in L and R ears: – sensitivity and specificity. -9/200 infants: L/R ear results in MR but missing in HS database. –13/200 infants: L/R ear results in HS database but missing in MR.

23. 23 Results – Reasons For No Screen Reason for no screen Character -istics HS MR % Missing In HS MR Sensitivity (%) (SE) Specificity (%) (SE) PVP (%) (SE) % refused 2.0 1.5 0 1.094.9 (6.3) 99.8 (0.2) 86.5 (15.1) % delayed 1.5 0 0 1.0NA98.9 (0.8) NA % equipment problem 1.5 0.5 0 1.010099.6 (0.3) 46.5 (4.2) % No reason given 6.5 8.6 0 1.01.4 (1.2) 97.9 (1.1) 3.2 (2.5)

24. 24 Results – Reasons for no screen specificity Variable sensitivity Variable positive predictive value In HS database, of 24/200 infants with missing L/R ear results… –Only 11 out of 24 with reason for why screen was missing or not done.

25. 25 Results – Birth and Screen Dates Discrepancies between dates Infant Birth Dates Hearing Screen Dates 0 days194129 1-3 days122 4-7 days00 8-14 days02 15-21 days02 22+ days03 Overall range-1 to 0-63 to 366

26. 26 Results – Birth and Screen Dates MR and HS: 3% of infant birth dates did not agree 29% of HS dates did not agree –Of the 58/200 discrepant screen dates: 22 missing in MR but available in the HS database 5 missing in the HS database but available in MR

27. 27 Noteworthy Findings Data fields with less frequent outcomes have lower validity: –Antibiotic use –Failed hearing screen results (left or right ear)

28. 28 Noteworthy Findings Low to moderate agreement among the various reasons for no hearing screen –But… small numbers = reduced precision Hospital-specific results

29. 29 Moderate agreement and high % missing among hearing screening dates –29% overall disagreement – 17% missing in medical records – 11% missing in hearing screening database. –Some screen dates in HS database recorded as being prior to birth date Noteworthy Findings

30. 30 Errors in abstraction process - Information recorded as missing by abstractors - Misinterpretation of language or results in medical record Medical record as gold standard assumption Study Limitations:

31. 31 Implement safeguards in hearing screening database Meet with hospitals Modify study design Where do we go from here?

32. 32 Minnesota Contact Information: Penny Hatcher, MSN, DrPH penny.hatcher@health.state.mn.usenny.hatcher@health.state.mn.us 651-201-3744 Yaoli Li, MD, MS, MA-CCC.A yaoli.li@health.state.mn.us 651-201-3750 Thanks You!