1 Ann Versporten, Ingrid Morales, Carl Suetens IPH, wednesday seminar: May 7, 2003 Scientific Institute of Public Health Data validation study of the National surveillance of nosocomial infections in intensive care units

2 Overview Background: overview national surveillance ICU Reasons for validation Validation study –Aims –Methods –Results Pneumonia Bacteraemia –Discussion –Conclusions –Recommendations

3 Background: National surveillance ICU 1996: Start National Surveillance of Hospital Infections (NSIH) : intensive care component (Pneumonia & Bacteraemia) –HELICS-based protocol (Hospitals in Europe link for Infection Control through Surveillance) –patient-based surveillance: 1 file by patient, + infection file if ICU-acquired PN or BAC –Nosocomial: infection acquired during hospital stay (admitted >48h in ICU)

4 Background: National surveillance ICU Objective: to follow-up nosocomial- infection rates Risk-adjusted infection rates are used as external benchmarks for comparison purposes

5 Methods: Data collection for ICU surveillance 1.Data at admission 2.Day-by-day e.g. central venous catheter, mechanical ventilation, antibiotic use 3.Infection data e.g. diagnostic criteria of PN, origin of BSI 4.Data at discharge

6 Reasons for validation Assessment of the validity of the findings Need to evaluate the accuracy of infection data reported to the NSIH program

7 Validation study

8 Main aim Validate reported ICU-surveillance data (ICU protocol: PN & Bac) against a reference gold standard Evaluate the accuracy of all data reported to the surveillance Evaluate the credibility of the surveillance

9 Specific aims Exhaustivity (completeness) denominator Sensitivity: probability of reporting a true PN & Bac to the ICU-surveillance Specificity: probability of reporting a PN & Bac as negative to the ICU-surveillance if the disease is truly absent Positive predictive value Negative predictive value

10 Accuracy of other protocol definitions SAPS II (Severity of illness score): estimate mortality in ICU-patients Workload data collection + input Factors which influence Sens. & Spec. Collection of data on anti-microbial resistence

11 Methods - 1 Sampling of hospitals:  Systematic sampling of 45 hospitals on the base of a list of hospital-trimesters (ICU participation period 01/01/1997 – 31/12/1999) Replacement: later period accepted Informed consent, voluntary participation Retrospective chart review methodology

12 Methods - 2: Research program Sampling of patient files:  All reported PN+ & Bac+ (from surv.)  All records with a positive hemoculture reported on a laboratorium list (for all admitted patients on ICU) (estimation false-neg Bac)  A 20% random sample of the negative files (estimation of false-neg PN) Estimation of exhaustivity of denominator on the base of administrative lists of ICU-admissions

13 Methods - 3 Calculation Se, Sp and Predictive values  “gold standard” = research team Trained data collectors (IPH)  Application protocol definitions  validation: uniform & standardised  evaluation = blind  discrepant infections: reviewed by other colleague Confidential & anonymous treatment of patient data

14 Methods - 4 National results No individual hospital results, only discussion at end validation proccess  Quality of data  Questions

15 Results investigated patient files in analysis: pts staying >24h in ICU (23 hospitals) Infections reported by hospitals to surveillance:  147 Pneumonia  49 Bacteraemia Type of ICU: 91% polyvalent Size of ICU: mean 10 beds Length of stay: median = 4,7 days

16 Results - 2 Exhaustivity of denominators: –For all patients staying >24h in ICU  72,8% –For all patients staying >48h in ICU  81,2%

17 Results - 3: Pneumonia Results of validation study for PN (inf. file &/or dbd)

18 Results - 4: Bacteraemia Results of validation study for Bac (inf. file &/or dbd)

19 Results: SE & SP Se % (95% CI)Sp % (95% CI) Pneumonia Infection file 32,7 (25,2-41,2)98,5 (97,4-99,2) Inf.file &/or dbd 53,2 (43,5-62,7)98,5 (97,4-99,0) Bacteraemia Infection file 48,1 (29,2-67,6)99,3 (98,5-99,7) Inf.file &/or dbd 59,3 (39,0-76,9)99,1 (98,2-99,6)

20 Results: predictive values PPV (%)(CI)NPV (%)(CI) Pneumonia Infection file 78,6 (65,6-87,9)85,9 (83,5-87,9) Inf.file &/or dbd 79,6 (68,3-87,8)88,9 (86,8-90,8) Bacteraemia Infection file 65,0 (40,9-83,7)97,3 (96,0-98,2) Inf.file &/or dbd 65,3 (43,6-82,4)97,3 (96,0-98,2)

21 Discussion - 1 Exhaustivity denominator: improvement possible – risk of bias, e.g. if only high risk patients included Pneumonia: low Se., good Sp. Bacteraemia: low Se., good Sp.

22 Discussion – 2 Possible reasons for lack of sensitivity –30% of the results originate from 1997 (start surv. NI in ICU). –50% of the collected data correspond with the 3 first surveillance trimesters that hospitals participated to our ICU surveillance.  = Explanation of lack of accuracy in the interpretation of the protocol ?

23 Who are those missed patients ?? Why are there so many false negative Pneumonias ?

24 Characteristics false negative PN

25 Characteristics of false neg. PN Mortality similar to true+ PN but higher than true- PN Shorter stay compared with true PN Ventilation similar to true+ PN, but higher than true- PN Lower SAPS score compared with true PN Much higher PN risk score compared to pt without PN, but lower risk score than true positive PN

26 Characteristics of false neg. PN More often a PN whereby only +RX; without isolation of micro-org. Linked with lower internal quality of data False- PN: tendency of less good reporting of PN at end of the surveillance period compared with true+ PN (NS) Overall we can say that the False neg. PN resembles more to the true+ PN than the true- PN

27 Discussion: Post discharge? Post-discharge PN en Bac incorporated, but active surveillance after discharge of ICU not done by several hospitals (time consuming) Questionning hospitals – Bac: 50% does active surv. after discharge – PN: 39% does active surv. after discharge PN(inf+dbd): SE= 49,2; Sp=98,2 without PD Bac(inf+dbd): SE= 59,6; Sp=99,4 without PD Benefit of post-discharge in scoop of routine surveillance ?: less NI missed without post discharge! (Geffers, Gastmeier, et al. 2001; Hugonnet, Eggimann, et al. 2002)

28 Factors influencing the Se. & Sp. of the infection data Who collects data ? Who decides whether a PN should be reported or not ? Criteria of bloodculture? Adherence to protocol definitions Degree of workload (ratio pat.-staff) Size of hospital …

29 Conclusions Exhaustivity varies for each hospital, but remains satisfactory in general Bac more accurately reported than PN (Se) Seldomly infections reported which were not a nosocomial infection (Sp) Absence of a gold standard ! (problem for diagnostic of PN)

30 Conclusions (next) Establishing Se & Sp only possible at the end of validation study  Preliminary conclusions: Sensitivity rather low (identification of a NI through surveillance) Specificity is high (% files truly classified as non-NI)  Low Se has also been reported by the CDC: “The data collectors detected over 2,5 times as many PN,..” (Emori, Edwards, et al. 1998)

31 Recommendations Training of professionals in charge of surveillance (Ehrenkranz, Shultz, et al. 1995)  case definitions (e.g. PN-diagnostic: use of micro-biologic reports & AB-administration)  surveillance-methods Simplification of protocol Development of electronic surveillance

32 Recommendations (next) Validation on continuous basis  Training on the field  Optimalisation contacts IPH / hospitals