1. 43rd National Immunization Conference
Slide 1
Assessment of Multiple Data Sources Results in Increased Data Quality Standards in IIS
43rd National Immunization Conference
March 30, 2009
Nichole D. Lambrecht, MSc; Mike McPherson; Sue Bowden, RN; Kristin Shore; Mike Parsons; and Susan Dickman
Immunization Program
Kansas Department of Health and Environment
To protect the health and environment of all Kansans by promoting responsible choices.

2. Objectives Background of project and sources of data.
Slide 2
Background of project and sources of data.
Problems found from multiple data sources that cause poor data quality.
Tools developed to improve data quality.
Conclusions.
Give a quick background, run down the problems, describe the tools

3. KSWebIZ Background Opt-in state. Birth to death registry.
Slide 3
Opt-in state.
Birth to death registry.
KSWebIZ was deployed summer of 2005.
Vital Statistics import was in place at the beginning.
Approximately 414,000 patient records were loaded ( ).
Imports ~113 patients nightly (98% of birth cohort).
Started enrolling providers in Aug 2005.
Local health departments (LHDs) that would direct enter into IIS (included 42 of 105 LHDs).
Legacy data was extracted and then imported.
Private providers sites that had no electronic data.
Mention KSWebIZ has a fully functional inventory module. Birth cohort as of 2008 was 41, Explain that historical the most immunizations were given in the LHD setting so that is why they were first in line to be enrolled (rich in data).
Birth Cohort for 2007 = 41,951 (per 365 = 115) * 98% of imports = 113
2006 = 40,896; 112; 110
2005 = 39,701; 108; 106
2004 = 39,553; 108; 106

4. KSWebIZ Background (cont)
Slide 4
Medicaid HL7 batch interface was implemented in May 2007.
Approximately 133,000 new patients with ~1.5 million associated vaccines were added from legacy data ( ).
Nightly batches contain on average vaccinations.
Bi-directional, real-time HL7 interface developed by July 2007.
Two different vendors for LHDs in KS (63 of 105 LHDs).
Currently testing with vendors for private providers.
Medicaid daily estimates will be higher than monthly estimates because the monthly includes adjusted claims and might be less data.

5. Current Provider Sites
Describe where highest populated areas occur in the state.

6. Current Data Statistics
Slide 6
2007 US Census data - KS = ~2.8 million
KSWebIZ Total Number of Created Patients = ~1.4 million
Data from Stats by User type – needs updated. These are lists of created data. 1,402,959 patients as of March 2st.
2007 US Census data -Kansas population = ~2.8 million
Number of Patients = 1,366,832
Vital Imports = ~ 513,657
Medicaid Imports = ~129,110
HL7 new patients added = 339,040
Number of Vaccinations = 9,528,906
Historical Vaccinations = 8,824,229
Administered Vaccinations = 704,677
Medicaid Imports = 1,426,889
HL7 new vaccinations added = ,098,773
Statistics as of March 21st, 2009

7. Current Data Statistics
Slide 7
KSWebIZ Total Number of Created Vaccinations = ~9.8 million
Historical = 9,118,081
Administered = 720,261
Data from Stats by User type – needs updated. These are lists of created data. 9,838,342
Transition to next slide: given the number of data sources and high volume of interface data we have found data quality problems.
2007 US Census data -Kansas population = ~2.8 million
Number of Patients = 1,366,832
Vital Imports = ~ 513,657
Medicaid Imports = ~129,110
HL7 new patients added = 339,040
Number of Vaccinations = 9,528,906
Historical Vaccinations = 8,824,229
Administered Vaccinations = 704,677
Medicaid Imports = 1,426,889
HL7 new vaccinations added = ,098,773
Comment on we only have 50% of population – we have a long way to go.
Statistics as of March 21st, 2009

8. Problems Found Common among data sources Vital Incoming interface data
Slide 8
Common among data sources
Incoming interface data
If either SSN or DoB was off, even by 1 digit from already existing record then new patient gets created.
Data entry
Illegible documentation can lead to duplication.
Vital
Hep B birth dose
If exact administration date was not recorded in Vital system then the date defaults to date of birth.
Give

9. Problems Found (cont) Medicaid HL7
Slide 9
Medicaid
Not complete immunization picture – based on enrollment status with carrier.
Billing dates could differ from administration date by several days.
Denied vaccinations erroneously sent to us.
Coding errors.
May bill for Hibtiter when it was actually HibPedvax that was administered.
HL7
Incorrect CVX codes lead to duplication.
During pre-certification phase duplicate vaccines were identified from provider’s database.
Separate databases require ongoing synchronization efforts to maintain high data integrity.
Transition to next slide: So how did we solve these problems?
Dup: Same date different code, same antigen within 10 days apart

10. Resolution Path
Slide 10
Evaluation of data sources needed to take place based on:
type of connection
timeliness of exchange
type of data
Overall ranking based on source and quality.
So how did we solve these problems? We ranked the source based on reliability of “correctness”.
Reference the MIROW docs – vaccine de dup and data quality chapters

11. Data Quality Considerations
Slide 11
Things to consider when ranking.
Patient level (demographics):
What was the source (encounter, billing, vital).
What mechanism was the data input to IIS (direct entry, one time data load, interface).
How often was it updated (as the patient was seen, nightly, weekly).
Vaccine level:
All listed above.
Who administered the dose (lot #, etc. improves rank).
If historical, did more than one provider confirm the date?
Interface = batch or two-way

12. Evaluation of Data Sources
Slide 12
Source – encounter, billing, vital
Vital – highest level of reliability for demographic data but not necessarily for vaccine level (Hep B birth dose).
Medicaid – medium level of reliability for demographic data but low on vaccine level.
Encounter – can depend on individual provider but generally the most reliable outside vital patient demographics.
HL7 –depends on vendor type since we are at the mercy of the EMR data standards.
Direct entry – generally most reliable of encounter data because we can impose data entry standards.
MMIS reason medium = address is usually most recent but spelling of name and SS# could be typos.

13. Evaluation of Data Sources (cont)
Slide 13
Mechanism – direct entry, one-time load, interface
Direct entry – most reliable since this occurs at individual patient entry and uses looser search criteria allows for near matches to display.
Interface (real time or batch) – medium reliability since this occurs at individual patient entry but utilizes strict matching criteria.
Legacy Load – lower reliability since individual records are not verified and data does not go through same matching criteria as interface data.

14. Evaluation of Data Sources (cont)
Slide 14
Timeliness of update – real time, nightly, weekly
Vital – imported nightly (95% within 6 days of birth).
Medicaid – imported nightly.
Real-time interfaces – as they see the patient (generally).
Direct Entry – as they see the patient.
Vaccine ownership – administered vs. historical
If administered by a provider they have highest reliability.
If entered as historical lower priority than administered but higher than billing data.
Multiple providers agreement on a date.
Timeliness is important because if not de-duplicated with real time data then it can compound the issue. The Real-time interface also can be different in the type of interface (automatic update or manual push of a button).

15. Patient De-Dup Matrix VITAL Medicaid HL7 DIRECT ENTRY (0) Vital
Slide 15
VITAL
Medicaid
HL7
DIRECT ENTRY
(0)
Vital
Vital (1)
Most current vaccine
Most current vaccine (2)
Most current vaccine (4)
Most current vaccine (3)
DIRECT ENTRY
(0) Very rarely found. Need to check Vital interface for problems.
(1) We keep Vital since we know these are correct demographics. Share any changes with HL7 provider if their record was modified (E.g. DoB, SSN).
(2) If HL7 record changed contact provider to make changes on their end.
(3) If different HL7 providers take most current vaccine or who administered. If same HL7 provider, contact them to make correction on their end too.
(4) If direct entry kept, then HL7 provider must be made aware of changes so that they can correct on their end.
When we get to this point we have conducted all the automatic corrections possible the duplicates that go through this matrix are ones that need human intervention.

16. Vaccine De-dup Script
Slide 16
A script was written to pull patients with vaccines in the same antigen group within 10 days of each other.
The query displays with the following information:
Patient info (system id, name, DOB).
Vaccine 1 & 2 (type, date, provider, user that create the record, and user that last updated).
So once we have this excel sheet then how do we handle or sort things. We have to apply rules of engagement.
HL7-1 = This is the provider that the duplicate vaccine report was run on.
HL7-2 = Another HL7 provider, other than HL7-1 that the report was run on.
OTH = Direct entry provider.
PR = Entered as a historical shot.
Provider, Provider = Clinic code and Created By fields
Provider, Provider, Provider = Clinic code, Created By, and Last Updated By fields

17. Vaccine De-dup Matrix
Slide 17
This is just an example of how complex this work can be.
Legend
HL7-1 = This is the provider that the duplicate vaccine report was run on.
HL7-2 = Another HL7 provider, other than HL7-1 that the report was run on.
OTH = Direct entry provider.
PR = Entered as a historical shot.
Provider, Provider = Clinic code and Created By fields
Provider, Provider, Provider = Clinic code, Created By, and Last Updated By fields

18. Vaccine De-dup Matrix Applied
Slide 18
Resolution: HL7 Provider 1 (Allen County) needs to verify paper records and correct in their system and then we need to simultaneously correct on our side also.
Example is dup strictly from same HL7 provider. Explain if you hover over the green question mark then you can see who created and last updated the record. Some scripts have been developed off the excel file to weed out easy ones like this. So then a list is developed for these type of providers and sent so they can let us know which one is correct or not. We have also now updated our pre-certification process to check for these prior to their load into production. At the time this provider was loaded the check was not in place so now it takes manual cleanup.
HL7-1 = This is the provider that the duplicate vaccine report was run on.
HL7-2 = Another HL7 provider, other than HL7-1 that the report was run on.
OTH = Direct entry provider.
PR = Entered as a historical shot.
Provider, Provider = Clinic code and Created By fields
Provider, Provider, Provider = Clinic code, Created By, and Last Updated By fields

19. Vaccine De-dup Matrix Applied (cont)
Slide 19
Resolution: Have HL7 Provider 2 (Crawford County) correct their date to reflect date of HL7 Provider 1 (Allen) since it was administered and then delete PR out of registry.
Legend
HL7-1 = This is the provider that the duplicate vaccine report was run on.
HL7-2 = Another HL7 provider, other than HL7-1 that the report was run on.
OTH = Direct entry provider.
PR = Entered as a historical shot.
Provider, Provider = Clinic code and Created By fields
Provider, Provider, Provider = Clinic code, Created By, and Last Updated By fields

20. Results Automated scripts run regularly.
Slide 20
Automated scripts run regularly.
Vital script – remove duplicate Hep B doses.
Medicaid script – remove any duplicate Medicaid vaccines.
Matrices (with associated dup scripts) developed and implemented.
Tightened the pre-certification protocol for interface providers that include duplicate patient and vaccine checks prior to go live.
Developed new reports for direct entry providers to identify their own duplicate patients and vaccines so they can proactively cleanup prior to IIS DQ checks.
Future to send MMIS facility so we know who to contact for errors. Vital - that match up against any other Hep B vaccine within 5 days. MMIS - that is within 5 days of another vaccine in same antigen group.

21. Conclusions
Slide 21
New and improved ways to check data quality need to be constantly considered.
Evaluation of data sources may create universal data quality standards and tools to improve system integrity.
As provider education increases so does need for data quality standard.
Data quality is constantly changing and as new data sources and technologies are available new and improved ways to check data quality need to be considered.
Although data source evaluation may differ among individual IIS overall the evaluation of data sources connected to the IIS may create universal data quality standards and tools to improve system integrity.

22. Contact Details Nichole Lambrecht, MSc
Slide 22
Nichole Lambrecht, MSc
Kansas IIS (KSWebIZ) Project Manager
Kansas Department of Health & Environment
Phone:
Website: