1. Washington State’s Approach: From CLABSI to SSI CSTE Annual Conference, 2013 David Birnbaum, PhD, MPH

2. Overview Now in our 4 th year for CLABSI validation Two components in our annual protocol –Both draw samples that span at least one entire year of surveillance in each hospital. –Annually, the first component reaches all (62) hospitals in our reporting network –The second reaches a number that we do not disclose Confidential collaboration in each hospital –“quality improvement” confidentiality laws

3. Overview (cont.) “Internal” self-assessment component –Every hospital in our reporting network follows our instructions once annually to conduct its own “internal” assessment of CLABSI surveillance program sensitivity & specificity Takes each hospital ≤6 hours per year Takes us approx. 30-60 minutes to review all the results each year

4. Overview (cont.) “External” validation verification visit component –Hospitals selected on basis of questionable “internal” results or a spot-check basis –ISO-2859 double-sampling technique used Gives sample size for defined statistical power to detect any performance levels below those we specified as acceptable –β-error (consumer ‘s risk) has more emphasis than α-error (producer‘s risk) in acceptance sampling Takes 1 qualified person ≤1 day per hospital to conduct the review That equates to less than 0.5 FTE to run our CLABSI validation work Anticipate when we begin SSI validation it will be a similar staffing workload

5. Overview (cont.) During our visit, we check to see that: –various practices are in place –sensitivity & specificity for detection and review of bloodstream infection cases meets our standards –coding of CLABSI cases (if any) conforms to NHSN criteria and meets our standards –any discordance between clinical (ICD-9 billing proxy) versus surveillance (NHSN) coding can be reasonably explained We also estimate total CLABSI burden –in-scope vs. out of scope hospital-acquired infections –infections present on admission –total range spanning a minimum from the restrictive NHSN criteria to a maximum from the more permissive ICD-9/clinical criteria using capture-recapture formulae

6. Technical aspects consistent with other industries We validate accuracy of the surveillance & reporting supply chain process, not “data” We view this as a quality control type problem, not an estimation type problem –More like HACCP than one-time studies… –Actively manages process quality, introducing quality improvement when appropriate –Ensures hospitals maintain a set standard for sensitivity & specificity that we defined

7. Device day check in data cleaning (not in validation visit) Brushed scatterplot matrix, highlighting highest number of central line days

8. Definition of Technical Terms Definition from A Dictionary of Epidemiology, 5 th Edition (International Epidemiological Association) –Validation – “The process of establishing that a method is sound.” –Data Cleaning – “The process of excluding the information in incomplete, inconsistent records or irrelevant information…before analysis begins…” Definitions from Quality Glossary, American Society for Quality, available at http://asq.org/glossary/http://asq.org/glossary/ –Validation – “The act of confirming a product or service meets the requirements for which it was intended.” –Verification – “The act of determining whether products or services conform to specific requirements.” –Acceptance sampling – “Inspection of a sample from a lot to decide whether to accept that lot. There are two types: attributes sampling and variables sampling. In attributes sampling, the presence or absence of a characteristic is noted in each of the units inspected. In variables sampling, the numerical magnitude of a characteristic is measured and recorded for each inspected unit; this involves reference to a continuous scale of some kind.”

9. Definitions (cont.) –Acceptance sampling plan – “A specific plan that indicates the sampling sizes and associated acceptance or nonacceptance criteria to be used. In attributes sampling, for example, there are single, double, multiple, sequential, chain and skip-lot sampling plans. In variables sampling, there are single, double and sequential sampling plans. For detailed descriptions of these plans, see the standard ANSI/ISO/ASQ A3534-2-1993: Statistics—Vocabulary and Symbols— Statistical Quality Control.” –Audit – “The on-site verification activity, such as inspection or examination, of a process or quality system, to ensure compliance to requirements. An audit can apply to an entire organization or might be specific to a function, process or production step.” –Process – “A set of interrelated work activities characterized by a set of specific inputs and value added tasks that make up a procedure for a set of specific outputs.”

10. Definitions (cont.) Definitions from HF-2 Business Operating Systems (BOS) for Health Care Organizations: Requirements for Process Improvements to Achieve Excellence. 2007, AIAG, Michigan –Validation – “objective confirmation that the specified requirements for a specific use or application have been satisfied in the end-user environment” –Verification – “objective confirmation that the specified requirements have been satisfied”

11. Immediate Priority Issue! Validate the Accuracy of What is Reported –Our industry too often uses a sad disclaimer “Resources for this audit were limited to a review of 200 patient records. Determination of an appropriate sample size is difficult and the sample size (202) is too small to draw statistically significant conclusions about the validity of CLABSI data reported to NHSN.” –Other industries use international standards (ISO 2859, ANSI/ASQ Z1.4, MIL STD 105E)

12. Validation in Other Industries Acceptance sampling uses statistical sampling to determine whether to accept or reject a production lot of material. It has been a common quality control technique used in industry and particularly the military for contracts and procurement. –1930s Dodge-Romig acceptance sampling tables –1940s MIL STD 105A –1990s MIL STD 105E cancelled, ANSI/ASQ Z1.4 in U.S. ISO 2859 as international equivalent

13. Our Operating Characteristics Curves See our technical reference manual –Examples on the next two slides First slide is from “internal” component Second slide is from “external” component’s double-sampling plan (at “normal” inspection level) –40 positive blood culture charts »Read 20; pass or fail or read remaining 20 –Also review all charts tagged with ICD9 code 999.31 –Reference manual available upon request

16. Future Plans Introduce SSI reporting (August 2012) –CMS requires abdominal hysterectomy & colon resection –State law requires abdominal & vaginal hysterectomy, total hip, total knee & open chest cardiac –We hope state law can be amended in the next session to match federal requirements Pilot test SSI validation (2012-2013) Publish SSI 2013 annual rates (2014) –Publish 2012 (partial year?) rates (2013) Implement SSI annual validation (2014)

17. Future Plans (cont.) What we plan to report on public website –Questions patients can ask and practices to watch for –SSI rates for specified surgical procedures How we plan to display this information –Incidence rate by procedure (map, table & box plot formats) Rates shown separately for complex vs. superficial SSI –Whether SIR value is 1 But not display the actual SIR number Thoughts thus far on validation protocol –100% compliance with surveillance practices –95% sensitivity (+5/-10 tolerance) & 98% specificity –Same two component & ISO 2859 approach as used for CLABSI

18. Expect all SSI surveillance programs to include the following activities: Reviewing in-patients with positive microbiology results from surgical site wound, abscess, aspirate or tissue, or blood cultures, post-operatively – at least once a week if not daily Reviewing cases with an ICD-9 code suggestive of surgical site infection upon any readmissions – at least weekly to monthly (a post-discharge surveillance activity already present in 95% of facilities according to our survey*) Communicating with ICP of other hospitals or with clinical lead at an ambulatory surgery facility when patients with surgical site infection coded as present on admission are admitted following surgery at another facility – at least weekly to monthly Communicating with surgeons, primary care physicians, surgical patients, or a combination thereof to confirm post-operative status several weeks after discharge – at least monthly to quarterly (63% of facilities already exchange lists with surgeons, 18% telephone patients, 8% have patients return a post-card, according to our survey*) *Zarate R, Birnbaum D. Postdischarge Surgical Site Infection Surveillance Practices in Washington Acute Care Hospitals. INFECT CONTROL HOSP EPIDEMIOL 2012;33(1):87-89.

19. Expect all SSI surveillance programs to include the following activities: New thoughts in November 2012 Reviewing in-patients with positive microbiology results from surgical site wound, abscess, aspirate or tissue, or blood cultures, post-operatively – at least once a week if not daily Maybe also imaging re: abscess after colon surgery Reviewing cases with an ICD-9 code suggestive of surgical site infection upon any readmissions – at least weekly to monthly (a post-discharge surveillance activity already present in 95% of facilities according to our survey*) Review either ICD-9 & CPT codes (Harvard list), or NSQIP SSI cases instead Communicating with ICP of other hospitals or with clinical lead at an ambulatory surgery facility when patients with surgical site infection coded as present on admission are admitted following surgery at another facility – at least weekly to monthly Communicating with surgeons, primary care physicians, surgical patients, or a combination thereof to confirm post-operative status several weeks after discharge – at least monthly to quarterly (63% of facilities already exchange lists with surgeons, 18% telephone patients, 8% have patients return a post-card, according to our survey*) *Zarate R, Birnbaum D. Postdischarge Surgical Site Infection Surveillance Practices in Washington Acute Care Hospitals. INFECT CONTROL HOSP EPIDEMIOL 2012;33(1):87-89.

20. The OC curve describes the probability of accepting a lot, OC(p), as a function of the proportion non-conforming (p), for the single and double sampling plans described above. The ASN curve describes the average sample size when employing double sampling, as a function of the proportion of non-conforming items (p). Sensitivity 95%Sensitivity 90%Sensitivity 85%Sensitivity 80%Sensitivity 70%Sensitivity 60% Cases Individual Probability Cumulative Probability Individual Probability Cumulative Probability Individual Probability Cumulative Probability Individual Probability Cumulative Probability Individual Probability Cumulative Probability Individual Probability Cumulative Probability 0 0.00000 1.000000.000001.000000.000001.000000.000001.000000.000001.00000 1 0.00000 1.000000.000001.000000.000001.000000.000001.000000.000001.00000 2 0.00000 1.000000.000001.000000.000001.000000.000001.000000.000001.00000 3 0.00000 1.000000.000001.000000.000001.000000.000001.000000.000011.00000 4 0.00000 1.000000.000001.000000.000001.000000.000001.000000.000070.99999 5 0.00000 1.000000.000001.000000.000001.000000.000011.000000.000350.99992 6 0.00000 1.000000.000001.000000.000001.000000.000040.999990.001500.99957 7 0.00000 1.000000.000001.000000.000001.000000.000200.999960.005130.99808 8 0.00000 1.000000.000001.000000.000011.000000.000880.999750.014420.99295 9 0.00000 1.000000.000001.000000.000050.999990.003200.998870.033640.97853 10 0.00000 1.000000.000021.000000.000280.999940.009710.995670.065600.94489 11 0.00000 1.000000.000100.999980.001240.999650.024710.985960.107340.87929 12 0.00000 0.000021.000000.000530.999880.004550.998410.052850.961260.147600.77195 13 0.00000 0.000130.999980.002310.999350.014000.993860.094860.908400.170310.62435 14 0.00001 0.000730.999850.008420.997040.036000.979860.142290.813540.164220.45405 15 0.000060.000070.003510.999120.025450.988610.076810.943860.177070.671250.131380.28982 16 0.000510.000580.013830.995610.063110.963160.134410.867050.180760.494180.086220.15844 17 0.003440.004020.043920.981780.126210.900050.189760.732640.148860.313410.045640.07223 18 0.018160.022180.109790.937870.198660.773840.210840.542880.096490.164550.019020.02658 19 0.072640.094820.208030.828070.237000.575180.177550.332040.047400.068060.006010.00756 20 0.207030.301850.280840.620040.201450.338180.106530.154490.016590.020670.001350.00156 21 0.374620.676470.240720.339200.108720.136720.040580.047960.003690.004080.000190.00021 22 0.323531.000000.09848 0.02800 0.00738 0.00039 0.00001

21. Remaining Questions 1.What microbiology triggers to specify 1.Specimen types & organisms to trigger case review by ICP for possible SSI 2.What ICD-9 triggers to specify 1.How many and which codes to trigger case review by ICP for possible SSI 3. If/When to add post-op non-prophylactic antimicrobial usage as a trigger 4. Whether to assess process measures in validation (e.g. SCIP measure data)

22. Remaining Questions New thoughts in November 2012 1.What microbiology triggers to specify 1.Specimen types & organisms to trigger case review by ICP for possible SSI Consider Laboratory Messaging Community of Practice (LabM CoP) group’s documents to identify synonyms & electronic coding conventions 2.What ICD-9 triggers to specify 1.How many and which codes to trigger case review by ICP for possible SSI Obtain list of ICD-9 & CPT codes from Harvard group 3. If/When to add post-op non-prophylactic antimicrobial usage as a trigger 4. Whether to assess process measures in validation (e.g. SCIP measure data)