1. Quality of Electronic Emergency Department Data: How Good Are They?
Trang Q. Nguyen, MD, MPH Michael Medvesky, MPH
New York State Department of Health

2. Need for Evaluation of the NYS Asthma ED Data
First time electronic ED data are available statewide
Potentially use in the state asthma surveillance system: to target, plan, monitor, and evaluate asthma interventions
Quality of information is unknown
No validation performed to date
Interests in diagnoses and socio-demographic information

3. Research Question
Is the NYS electronic ED data for asthma & respiratory visits (diagnosis and socio-demographic information) comparable to the original ED chart data, so that it can be used in the State Asthma Surveillance System?

4. Methodology
Inclusions: ED records with diagnosis of asthma (ICD-9 CM 493) or respiratory disease (ICD-9 CM excl 493)
Types of records included:
Type 1: asthma as primary diagnosis;
Type 2: asthma as other diagnosis;
Type 3: other respiratory illness (excluding asthma) as a primary diagnosis

5. Methodology (cont’d) Hospital sampling scheme: Chart sampling scheme:
October 1, 2003 to September 30, 2004 (to control for seasonality) this group included hospitals that voluntarily submitted ED data (n=140)
For each sample, a random group of hospitals was selected
Chart sampling scheme:
For each selected hospital, ED records for three types of diagnoses were randomly selected
Each chart was weighted based on the number of selected charts and the total number of each type of diagnosis in that specific hospital

6. Methodology (cont’d)
For each ED record sampled, the hospital was asked to send photocopies of the ED face sheet, ED assessment form and ED discharge summary to NYSDOH
Variables collected included: medical record number, admitting diagnosis/patient reason for visit, principal/primary diagnosis, other diagnosis, admission date, patient gender, patient date of birth, patient race, patient ethnicity, patient zip code, source of payment, patient status at disposition
These ED chart variables were reviewed, coded and entered into an electronic database, and then matched and compared with electronic ED data

7. Methodology (cont’d) Types of analyses:
Positive predictive value (PPV) of each diagnostic category
Sensitivity* of each diagnosis category
Percent of records matched by patient age, gender, race, ethnicity, patient zip code, source of payment, and patient status at disposition between electronic and chart data
PPV and sensitivity analyses by hospital volume
* Assuming the number of the ED charts that did not get entered into electronic ED database was insignificant.

8. Results

9. Results of Data Collection:
1,360 charts received, 2.4% missing
Sample
Number of Hospitals
Number of Charts
Requested
Received
(Missing)
1st Sample 40
548
540 (8)
2nd Sample 35 34
464
445 (19)
3rd Sample 32
381
375 (6)
University IRB approval
Obtain de-identifiable matched ED data from the Public Health Information Group: patient’s age, gender, race, ethnicity, zip code, source of payment and status at disposition information from both SPARCS ED records and ED charts

10. Estimates of Number of Records (Sample 1-3, Weighted Data)
ED Charts
Asthma PRDX
(n)
Asthma Other
Resp. PRDX (n)
Other PRDX (n)
Total
71,799
2,543
1,110
544
75,996
4,623
18,283
312
699
23,917
Resp. PRDX
1,136
2,240
229,760
5,046
238,183
Electronic ED

11. Summary of PPV for Asthma as Primary Diagnosis
Sample
Weighted %
95%CI 1
92.7 2
94.9 3
96.0
1-3
94.5
PPV for Asthma as Primary Diagnosis
There were very high PPVs (>90%) across samples and combined samples.
Combined sample 1-3, PPV was 94.5%, PPV ranged from 93-96% for individual samples
Combined sample 4-5: PPV was 91.6%. PPV ranged from % for individual samples
There was no significant variation of PPV between samples.
Majority of false positive charts were asthma as other diagnosis (53-60%), Page 9-sample1, page 16 –combined sample1-3, page 26-combined sample4-5

12. Summary of PPV for Asthma as Other Diagnosis
Sample
Weighted %
95%CI 1
93.4 2
70.5 3
98.5
1-3
76.4
PPV for Asthma as Other Diagnosis
There were mixed results of PPVs between samples.
Combined sample 1-3, PPV was 76.4%, PPV ranged from 71-99% for individual samples
Combined sample 4-5: PPV was 87%. PPV ranged from 82-93% for individual samples
There were greater variations of PPV between samples for both groups: voluntary reporting hospitals (sample 1-3) and non-voluntary reporting hospitals (sample -5).
However, these variations were not statistically significant.
Majority of false positive charts were asthma as primary diagnosis for sample 1-3 (page 16 –combined sample1-3), but not for sample 4-5 (page 26-combined sample4-5)

13. Summary of PPV for Respiratory Illness as Primary Diagnosis
Sample
Weighted %
95%CI 1
96.3 2
96.8 3
1-3
96.5
PPV for Respiratory Illness as Primary Diagnosis
There were very high PPVs (>90%) across samples with the exception of sample 4 (72%).
There were consistent values of PPV for the first group of hospitals. Combined sample 1-3, PPV was 96.5%, PPV ranged from 96-97% for individual samples.
For the second group of hospitals, there were significant difference of PPV between sample 4 and 5. PPV for sample 4 was 72% while this was 92% for sample 5. The combined sample 4-5: PPV was 91.6%.
Overall, there was a significant variation of PPV between the combined samples 1-3 and 4-5. This happed mainly due to the lower value of PPV in sample 4.
Majority of false positive records were neither asthma as primary diagnosis nor asthma as other diagnosis for the combined sample 1-3 ( page 16). For the combined sample 4-5, however, there was a large portion of the false positive records felt into asthma as other diagnosis (page 26).

14. Summary of Sensitivity for Asthma as Primary Diagnosis
Sample
Weighted %
95%CI 1
98.4 2
85.5 3
96.2
1-3
92.6
Sensitivity for Asthma as Primary Diagnosis
There were very high sensitivity (>90%) across samples with the exception of sample 2 (85.5%).
For the first group of hospitals, sensitivity was 92.6% for the combined sample 1-3, sensitivity ranged from % for individual samples. Sensitivity from sample 2 was significantly lower than that for sample 1.
For the second group of hospitals, sensitivity values were consistently very high for both samples (above 99%). This resulted in more than 99% for sensitivity of the combined sample 4-5.
Overall, there was a significant variation of sensitivity between the combined samples 1-3 and 4-5. This happed mainly due to the lower value of sensitivity in sample 2.
Majority of misclassified charts (>80%) were into asthma as other diagnosis. ( page 17 –combined sample 1-3, page 26-combined sample 4-5)

15. Summary of Sensitivity for Asthma as Other Diagnosis
Sample
Weighted %
95%CI 1
69.2 2
81.6 3
80.4
1-3
79.3
Sensitivity for Asthma as Other Diagnosis
There were great variation of sensitivity estimates across samples, sensitivity ranged from 38.7% (sample 4) to 97% (sample 5).
For the first group of hospitals, insensitivity estimates were in the moderate range (70%-82%). The was no significant variation in sensitivity estimates
For the second group of hospitals, sensitivity values varied a great deal. Sensitivity of sample 4 was 39% with 95%CI [ ] and sample 5 was 97% with 95% CI [ ]. This variation was statistically significant and the reason for this variation remained unknown. We could go back and revalidate for sample 4 with regard to this type of diagnosis.

16. Summary of Sensitivity for Respiratory Illness as Primary Diagnosis
Sample
Weighted %
95%CI 1
98.9 2
99.8 3
99.5
1-3
99.4
Sensitivity for Respiratory Illness as Primary Diagnosis
There were almost perfect sensitivity estimates (>98%) across samples.
For the first group of hospitals, sensitivity was 99.4% for the combined sample 1-3, sensitivity ranged from % for individual samples.
For the second group of hospitals, sensitivity values were also consistently very high for both samples (above 98%). This resulted in nearly 99% for sensitivity of the combined sample 4-5.
Overall, there was no variation of sensitivity between single samples or the combined samples 1-3 and 4-5.

17. Summary of % Matched for Demographic Variables
% Matched
95%CI
Age (+ 2)
96.6
Gender
98.8
Race
57.2
Payment Source
56.7
Zip code
93.3
Disposition
97.2
The next 2 slides represent the summary of % matched for demographic variables for the two combined samples 13- and 4-5.
There were no or little missing data with regarding the age variable for the two combined samples 1-3 and 4-5 in both SPARCS and chart data (page 17 and 27). Overall, there were very high percents of matched information for patient’s age between SPARCS and chart data (nearly 97% for both samples). For our surveillance purpose, age data matched within 2 years could be acceptable.
Similar to the age variable, there were no missing data for gender variable in both SPARCS and chart data. Percent matched for both combined samples were nearly 99% (page 17 and 27).
Race variable, however, had some unknown/missing data (4%) from the SPARCS data for both combined samples. The proportions of missing data from the chart source even a lot higher comparing to the SPARCS, 24% and 35% for the combined samples 1-3 and 4-5, respectively. Of the non-missing data, there were not great percent of matched data between the two sources. This happened consistently in both combined samples (57% and 50% for the combined samples 1-3 and 4-5, respectively). Since information for this variable were often missing from the chart source, it was possible that hospitals used other methods of collecting this information and added into the data before they would submit to the state.

18. Distribution of Hospitals in the Collected Samples by ED Volume
Hospital ED Volume
Number (%)
Low
9 (14.0)
Medium
22 (34.4)
High
33 (51.6)
This table presents the number of hospitals that were selected for each combined sample stratified by the ED volume. For the combined sample 1-3, over a half of hospitals were in the high volume category and only 14% of hospitals were in the low volume category.
For the combined sample 4-5, however, 42% of the hospital were in the low volume category and 25% were in the high volume category.
ED volume based on the total number of outpatient ED visits from 2005 data for all ED hospitals in the NYS outpatient dataset. (Low volume: first and second quartiles- less than 17,968 ED records in 2005; Medium volume: third quartile - between 17,968 and less than 34,580 ED records in 2005; and High volume: fourth quartile - more than 34,580 ED records in 2005)

19. Summary of PPV by Hospital ED Volume – Combined Sample 1-3
ED Volume
Weighted %
95%CI
Asthma as Primary Diagnosis
Low
97.3
Medium
91.7
High
95.1
Asthma as Other Diagnosis
94.6
93.5
71.1
Respiratory Illness as Primary Diagnosis
97.1
95.9
96.6
This table shows the summary of PPV by hospital ED volume for the combined sample1-3. ED volume was stratified using the total number of outpatient ED visits from 2005 data for all ED hospitals in the SPARCS outpatient dataset. (Low volume: first and second quartiles- less than 17,968 ED records in 2005; Medium volume: third quartile - between 17,968 and less than 34,580 ED records in 2005; and High volume: fourth quartile - more than 34,580 ED records in 2005).
Overall, hospitals with low volume had the highest PPV estimates for all three types of diagnoses. For asthma as primary diagnosis – type 1, and respiratory illness as primary diagnosis – type 3, there was no significant variation of PPV by the volume of patients visiting the ED. For asthma as other diagnosis (type 2), however, there was a significantly lower PPV for group of hospitals with high ED volume when compared to the group of hospital with medium ED volume (71.1%, 95%CI [ ], and 93.5%, 95%CI [ ], respectively).

20. Summary of Sensitivity by Hospital ED Volume – Combined Sample 1-3
ED Volume
Weighted %
95%CI
Asthma as Primary Diagnosis
Low
100.0
Medium
99.6
High
90.7
Asthma as Other Diagnosis
86.0
92.3
75.4
Respiratory Illness as Primary Diagnosis
99.7
99.1
99.4
This table shows the summary of sensitivity by hospital ED volume for the combined sample1-3.
Overall, hospitals with low volume had the highest sensitivity estimates for two of the three types of diagnoses (type 1 and type 3). For type 1, there were significantly higher sensitivity estimates among hospitals with low and medium ED volume when compared to hospitals with high ED volume. For type 3, sensitivity estimates were consistently high (more than 99%) across all hospitals regardless of ED volume. There was mixed results for type 2, however, this variation was not statistically significant.

21. Limitations
Information on % of ED charts that did not get entered (missing) into the electronic ED database is unknown.
Did not select charts with other non- respiratory diagnoses. Therefore, unable to decide the % of misclassification of asthma into other non-respiratory diagnoses.

22. Conclusions
Electronic ED data quality is very good for primary diagnoses for asthma and respiratory illness good for use in surveillance
Quality of asthma as other diagnosis not as good
Demographic information: age, gender, ZIP code, and source of payment are good could be used for asthma, respiratory and other diseases/conditions surveillance
Race and ethnicity information are unreliable
Hospitals with high ED volume had tendency of having low data accuracy  recommend to improve data collection

23. Thank you !

24. Methodology (cont’d) Validation Process for Diagnoses
If ICD 9 was present on the ED chart  record on validation sheet
If ICD 9 was not coded on the ED chart  use chart narratives and code book to determine the ICD 9 diagnosis for entry
Past medical history of asthma was frequently categorized as “other diagnosis”

25. ED chart - Gold Standard+ -
Electronic
ED data a b
a+b c d
c+d
a+c
b+d n
__a__
a + c
Sensitivity=
__a__
a + b
Positive Predictive Value=

26. Percent of Each Category of Charts by SPARCS Category (Sample 1-3, Weighted Data)
ED Charts
Asthma PRDX
(%)
Asthma Other
Resp. PRDX (%)
Other PRDX (%)
Total
94.5
3.3
1.4
0.7
100
19.3
76.4
1.3
2.9
Resp. PRDX
0.5
0.9
96.5
2.1
SPARCS ED

27. Percent of Each Category of SPARCS by Chart Category (Sample 1-3, Weighted Data)
ED Charts
Asthma PRDX
Asthma Other
Resp. PRDX
Asthma PRDX (%)
92.6
11.0
0.5
Asthma Other (%)
6.0
79.3
0.1
(%)
1.4
9.7
99.4
Total (%)
100
SPARCS ED

28. Missing (Unknown) Information for Demographic Variables (Sample 1-3)
ED SPARCS
N (%)
ED Charts
Age
0 (0)
4 (0.3)
Gender
Race
58 (4.3)
327 (24.0)
Payment Source
123 (9.0)
Zipcode
6 (0.4)
Disposition
13 (1.0)

29. Other Demographic Variables-Comparing ED Charts with ED SPARCS (Sample 1-3, Raw Data)
N (%) Matched
N (%)
Non-matched
N (%) Missing
Age (+ 2)
1,326 (97.4)
31 (2.3)
4 (0.3)
Gender
1,343 (98.7)
18 (1.3)
Payment Source
847 (62.2)
391(28.7)
123 (9.0)
Zipcode
1,272 (93.5)
83 (6.1)
6 (0.4)
Disposition
1,330 (97.7)
13 (1.0)

30. Other Demographic Variables (Sample 1-3, Raw Data) - Cont’d
Matched
Non-matched
N (%) Missing
Race
797 (58.6)
254 (18.7)
310 (22.8)
12 hospitals of 70 selected hospitals(17%) in these samples did not collect race variable on the charts.
Ethnicity
Only one hospital in these samples did collect ethnicity variable on the charts.

31. Other Demographic Variables (Sample 1-3, Weighted Data)
% Matched
(SE)
% Non-matched (SE)
% Missing
Age (+ 2)
96.6 (0.77)
2.6 (0.67)
0.8 (0.42)
Gender
98.8 (0.46)
1.2 (0.46)
Race
57.2 (3.2)
13.3 (2.7)
29.5 (2.56)
Payment Source
56.7 (2.8)
31.0 (2.3)
12.2 (2.0)
Zipcode
93.3 (0.89)
6.2 (0.88)
0.5 (0.25)
Disposition
97.2 (0.67)
1.1 (0.36)
1.7 (0.59)