1. SIR 101: Interpretation and public reporting
Good afternoon and welcome to the SIR 101: Interpretation and public reporting webinar brought to you by the Virginia Department of Health’s Healthcare-Associated Infections Program. My name is Dana Burshell and I am the HAI Team’s Epidemiologist. I am happy to speak with you about the standardized infection ratio, also referred to as the SIR, and some of the ways it is being publicly reported. In addition, today we will also give you the first glimpse of the new VDH central line-associated bloodstream infection report which will start displaying the SIR in However, let us start with the place many people were first introduced to the SIR.
Dana Burshell, MPH, CPH, CIC
HAI Epidemiologist
Virginia Department of Health

2. NHSN’s Guide to the Standardized Infection Ratio (SIR)
The National Healthcare Safety Network (NHSN) is the Centers for Disease Control and Prevention’s online surveillance system for healthcare-associated infections. Knowing that this new SIR metric would be a significant change for healthcare professionals, NHSN published a special SIR edition in October of 2010 and updated it in December A link to the document online is found at the bottom of the slide. I highly suggest you print out and keep this document in your files because it reviews the SIR definition, benefits of the metric, how it is calculated for a central line-associated bloodstream infections (CLABSIs) and surgical site infections (SSI), how to interpret it, and what SIR options are available in NHSN reports. I have used this document to help frame the conversation and some of the examples will be found throughout the presentation. Let’s start with the definition.

3. What is a standardized infection ratio?
The standardized infection ratio (SIR) is a summary measure used to track healthcare-associated infections (HAIs) at a national, state, or local level over time. The SIR adjusts for patients of varying risk within each facility. - The National Healthcare Safety Network (NHSN)
What is the standardized infection ratio?
The SIR is a summary measure used to track healthcare-associated infections (HAIs) at a national, state, or local level over time. The measure adjusts for patients of varying risk within each facility.

4. Benefits of using the SIR
Single metric
One number that can be used to make comparisons
Scalable
National, regional, facility-wide, location-specific, by surgeon for SSIs, etc.
Can combine the SIR values at any level of aggregation
Can perform more detailed comparisons within any individual risk group
Risk-adjusted
Adjusts for factors known to be associated with differences in HAI rates
Risk-adjustment differs between types of HAIs and types of surgical procedures
The HHS HAI Action Plan succinctly summarizes the benefits of using the SIR. First of all, it is a single metric where one number can be used to make comparisons. Because the SIR is always compared to 1, it is relatively easy to assess performance. We will talk more about the interpretation of the SIR in the coming slides.
Secondly, the SIR is scalable because the SIR values at any level can be combined. It can be calculated at the national, regional, facility-wide, or location-specific levels. It can be calculated by surgeon or by procedure for surgical site infections. Even within each group, you can perform more detailed comparisons – a function that is made easier by using NHSN Reports.
The SIR is risk-adjusted for factors known to be associated with differences in HAI rates. The risk-adjustment may differ between types of surgical procedures, and the SIR remains flexible to any future changes to risk adjustment.
- HHS HAI Action Plan -

5. The SIR calculation
In HAI data analysis, the SIR compares the actual number of HAIs reported (observed) with the baseline U.S. experience (predicted) adjusting for several risk factors that have been found to be significantly associated with differences in infection incidence.
SIR is a ratio that is a comparison of two values
SIR = number of observed HAIs
number of predicted HAIs
The SIR is simply a comparison of two values: the actual number of HAIs reported (also known as what was observed) and the number of HAIs predicted for that population based on the baseline US experience. An easy way to think about it is observed over predicted.

6. What is the “baseline U.S. experience”?
NHSN aggregate data are used as the standard population and considered to be the baseline U.S. experience for the SIR calculations.
NHSN baseline data used in an SIR are used to calculate the predicted number of HAIs adjusting for the identified risk factors.
Currently, the SIR baseline US experience is based on the NHSN aggregate data. The NHSN baseline data are used to calculate the predicted number of HAIs. Note: This does not change from year to year. For example, 2009, 2010, 2011, and 2012 SIR calculations all use national data as their baseline. This is important when trending your data because, with a constant comparison group, you can track progress over time. If you use a different comparison group from year to year, your trend may not be as clear and may not be accurate.

7. What does the SIR number mean?
An SIR greater than 1.0 indicates that more HAIs were observed than predicted.
An SIR of 1.0 indicates that the number of HAIs observed was equal to the number predicted.
An SIR less than 1.0 indicates that fewer HAIs were observed than predicted.
However, the SIR alone does not imply statistical significance.
The SIR is only a point estimate and needs additional information to indicate if the finding is significant and not likely due to chance (that is, statistically significantly different from 1).
Accounting for differences in the types of patients and/or patient location:…

8. Statistical significance
A p-value and 95% confidence interval (CI) are calculated by NHSN for each SIR.
The p-value identifies if the information is statistically significant.
If the p-value is < 0.05, the SIR is “statistically significant".
The 95% CI can sometimes be used to approximate statistical significance.
A 95% CI assesses the SIR’s magnitude and stability.
If the SIR 95% CI does not contain the value 1, the SIR is considered "statistically significant".
Given the great diversity in patient populations, clinical services provided, and the small overall number of HAIs, statistical analyses are applied to ensure that appropriate comparisons are made between hospitals (MA 2012 report)

9. What does a significant SIR mean?
While in many cases, significantly high SIRs may reflect a need for stronger CLABSI prevention efforts and significantly low SIRs may support already existing strong CLABSI prevention efforts, several other factors such as validation of reported data may play a role. 
The real measure of success is following the SIRs over time to indicate if positive progress occurs and is sustained.
Because the ultimate goal is zero HAIs, prevention efforts are never complete.
If your SIR is significant, what does it mean?
While in many cases, significantly high SIRs may reflect a need for stronger CLABSI prevention efforts and significantly low SIRs may support already existing strong CLABSI prevention efforts, several other factors such as validation of reported data may play a role. 
The real measure of success is following the SIRs over time to indicate if positive progress occurs and is sustained.
Because the ultimate goal is zero HAIs, prevention efforts are never complete.

10. Explaining and interpreting the SIR: Virginia data
Time Period
CLABSIs observed (#)
CLABSIs predicted (#)
Central line days (#)
SIR
p-value
95% CI
2011
233
366
194,483
0.64
<0.001
0.56, 0.72
During 2011, there were 233 CLABSIs identified and 194,483 central line days observed in Virginia adult intensive care units.
Based on the NHSN baseline data and the composition of locations in Virginia facilities, 366 CLABSIs were predicted.
This results in an SIR of 0.64 (O/P= 233/366), signifying that during this time period, Virginia facilities identified 36% fewer CLABSIs than predicted.
The p-value (<0.001) and 95% confidence interval (0.56, 0.72) indicate that the number of observed CLABSIs is statistically significantly lower than the number of predicted CLABSIs. (Reminder: If the p-value is less than 0.05 and the 95% CI does not cross 1, the SIR is statistically significantly different than 1.)
NHSN output provides the following information: the time period, the number of observed CLABSIs, the number of predicted CLABSIs, the number of central line days, the calculated SIR and its p-value and 95% confidence interval.
With just this information, we can describe Virginia 2011 CLABSI data in a number of different ways.

11. Why publicly report HAI data?
Infection data can give healthcare facilities, patients, and public health agencies the knowledge needed to design and implement prevention strategies that protect patients and save lives.
Research shows that when healthcare facilities are aware of their infection issues and implement concrete strategies to prevent them, rates of certain hospital infections can be decreased by more than 70 percent. 
NHSN has explained why publicly reporting HAI data can be a positive action.
1) Infection data can give healthcare facilities, patients, and public health agencies the knowledge needed to design and implement prevention strategies that protect patients and save lives.
2) Research shows that when healthcare facilities are aware of their infection issues and implement concrete strategies to prevent them, rates of certain hospital infections can be decreased by more than 70 percent. 
Furthermore, Paul Levy, who was the former CEO of Beth Israel Deaconess Medical Center pointed out, “Transparency's major societal and strategic imperative is to provide creative tension within hospitals so that they hold themselves accountable. This accountability is what will drive doctors, nurses, and administrators to seek constant improvements in the quality and safety of patient care.”
- CDC NHSN 2009 Report Q&A:

12. Intra-facility data sharing benefits:
Top benefits identified by SSI pilot study IPs (2011)
Increased awareness of HAIs within the facility
Presented data to those who can make a difference
Provided benchmark data to support improvement initiatives
Kept HAIs in the spotlight
Infection preventionists participating in the 2011 surgical site infection surveillance pilot revealed some additional benefits of sharing data within the facility.
Increases awareness of HAIs within the facility
Presents data to those who can make a difference
Provides benchmark data to support improvement initiatives
Keeps HAIs in the spotlight
Although this is not specific to the SIR metric, it applies to all HAI metrics being shared within a facility.

13. How is the SIR being used currently?
NHSN SIR reports
CLABSI, SSI
CMS Hospital Compare website
CLABSI
Updated VDH HAI report
Other states
Within hospitals
The SIR is currently being used in a number of ways. NHSN has published SIR reports for CLABSI and SSI for 2009 including national and state-specific data. The Centers for Medicare and Medicaid Services (CMS) began publicly reporting CLABSI data in 2012 and will report SSI and catheter-associated urinary tract infection data in The VDH HAI Program has recently revamped their CLABSI report to include the SIR because we feel that the SIR is a more accurate metric than a rate when tracking HAI data at the hospital and state level. Many other states have also moved to using only the SIR including but not limited to New York, Tennessee, Maryland, California, and New Hampshire.
Because the SIR became available in NHSN in 2010 and is still relatively new to infection preventionists, facilities are just beginning to become familiar with the measure. We have been receiving an increasing amount of calls and s from infection preventionists asking for technical assistance prior to bringing the SIR to their staff. In just a few minutes, you will hear from infection preventionist Bonita Allen, who has just begun the education process of introducing the SIR to her facility.

14. NHSN SIR report: CLABSI July-Dec 2009
Virginia
US - all
Some of the Virginia-specific results from the NHSN CLABSI SIR Report July-December 2009 are shown here.
In the top table, we can see that there were 75 facilities reporting from Virginia, which was about 5% of all reporting US facilities.
During July-December 2009, there were 151 CLABSIs identified in Virginia adult and pediatric intensive care units.
Based on the NHSN baseline data and the composition of locations in Virginia facilities, 188 CLABSIs were predicted.
This results in an SIR of 0.80 signifying that during this time period, Virginia facilities identified 20% fewer CLABSIs than predicted.
The 95% confidence interval (0.58, 0.94) indicate that the number of observed CLABSIs is statistically significantly lower than the number of predicted CLABSIs.
Additionally, this report shows facility-specific SIRs at key percentiles. Half of Virginia facilities had an SIR of 0, the facility-specific SIR at 75% was 0.96 and at 90%, the SIR was As we would expect, there is a range in SIRs across Virginia acute care facilities.
This report also compares the SIR of the first and second halves of 2009 as seen in the bottom table. While the point estimate of Virginia decreased from 0.83 to 0.80, neither Virginia nor the US’s SIR changed significantly between the two time periods.
Note: Includes PICU, but not NICU
Note: Data only for states using NHSN to comply with a legislative mandate* to report HAIs to the state health department (reported as of September 2010)

15. Note: Reference Period = 2006-2008
In the same report, the overall US CLABSI SIR was graphed over time. Starting from a value of 1.00 during the reference period of , the SIR dropped to 0.82 at the beginning of 2009 and the point estimate increased slightly to 0.83 during the second half of 2009.
Note: Reference Period =

16. Hospital Compare Go to the Hospital Compare website
Find hospitals near your location
Choose up to 3 hospitals to compare
Select “Patient Safety Measures”
Scroll down to “Healthcare
Associated Infections (HAIs)”
Click “View Graphs”
CMS started collecting CLABSI data in 2011 and started publishing the results in 2012 on Hospital Compare. Hospital Compare displays various health quality report cards allowing comparisons of up to 3 hospitals at a time.
To get to the HAI section:
Go to the Hospital Compare website shown here on this screen shot
Find hospitals near your location
Choose up to 3 hospitals to compare
Select “Patient Safety Measures”
Scroll down to “Healthcare
Associated Infections (HAIs)”
Click “View Graphs”

17. Hospital Compare displays SIR
Only includes SIR point estimate (no p-value or 95% CI)
Language
Better than the US National Benchmark
Same as the US National Benchmark
Worse than the US National Benchmark
Not available
Lower numbers are better. A score of zero – meaning no CLABSIs – is best.
Hospital Compare chooses another way of displaying the data via a horizontal bar graph. Hospital Compare only reports SIR point estimate information and does not show p-values or 95% confidence intervals. The orange vertical line displays an SIR of 1, meaning the number of CLABSIs observed would the same as predicted. The yellow bar displays the state SIR of 0.60, and the 3 purple bars display 3 different hospital SIR point estimates.
Hospital Compare helps its consumers interpret the SIR:
A score of less than 1 means that the hospital had fewer CLABSIs than hospitals of similar type and size. Hospital Compare describes a hospital in this scenario as “Better than the US National Benchmark”.
A score of 1 means the hospital's CLABSI score was no different than hospitals of similar type and size. Hospital Compare describes a hospital in this scenario as “Same as the US National Benchmark”.
A score of more than 1 means the hospital had more CLABSIs than hospitals of similar type and size. Hospital Compare describes a hospital in this scenario as “Worse than the US National Benchmark”.
For some hospitals, the number of cases is too small to reliably tell how well a hospital is performing, and Hospital Compare describes this as “Not Available”.
Hospital Compare also points out that “Lower numbers are better.” and “A score of zero – meaning no CLABSIs – is best”.
If a hospital had no infections, Hospital Compare displays “This hospital’s score is 0”.

18. New VDH CLABSI report: Table 1 shows annual SIR of all hospitals
This is a glimpse of Table 1:Central line-associated bloodstream infections (CLABSI) annual report reported by Virginia hospitals with adult intensive care units, 2011 of the new VDH CLABSI report. You may notice a few major changes from the previous report:
There is no CLABSI rate
VDH initally used the CLABSI rate because it was the best metric available at the time. However, CLABSI rates could be potentially misleading.
There is an SIR and its corresponding data
Starting in 2012, the SIR will be used instead of the CLABSI rate because the SIR is a more accurate summary measure that helps to “level the playing field” by controlling for factors within hospitals that may place their patients at a greater infection risk. Presenting data as a SIR allows us to group data across risk categories, procedures, and hospitals to gain a better understanding of HAI occurrence while still adjusting for underlying patient or hospital factors.
It will be published annually, not quarterly
While hospital infection preventionists should look at their data on a monthly and quarterly basis to capture any areas of concern and make changes to their infection prevention program, on the state level, we think this report will be more useful showing annual data. Quarterly variations in the data may be confusing or too much information for some consumers; it is more likely that the general public will be interested in the overall SIR and annual trend. Also, more facilities will be able to populate an SIR with a longer time period. We wanted to maximize the number of facilities with an SIR.
There are colored circles
The colored circles are a guide to help easily interpret the SIR point estimate and its statistical significance.

19. New VDH CLABSI report: Color legend
In place of the use of the terms “statistically significant” or “statistically different” for the interpretation of the CLABSI SIR in this report, the following colors and symbols will be used.
Red - SIR indicates more infections observed than predicted (statistically significant)
Blue - SIR indicates a similar number of infections observed compared to predicted
Green - SIR indicates fewer infections observed than predicted (statistically significant)
NA – Not applicable. If fewer than one infection was predicted, the SIR is not calculated.

20. In Hospital X’s adult intensive care units during 2011, there were 5 CLABSIs identified and 2,611 central line days observed.
Based on the NHSN baseline data and the composition of locations in Hospital X, 5.3 CLABSIs were predicted.
This results in an SIR of 0.9 (O/P= 5/5.3), signifying that during this time period, Hospital X identified 10% fewer CLABSIs than predicted.
The 95% confidence interval (0.30, 2.21) indicates that the number of observed CLABSIs is not statistically different than the number of predicted CLABSIs.
Here is an example from Table 1.
The first row has overall state CLABSI data which includes all CLABSI data from acute care adults ICUs.
The next 3 rows contain a subset of this CLABSI data stratified by bedsize category. There are 3 bedsize categories: <=200, , >500
The subsequent rows contain the individual hospital’s CLABSI data, and they are ordered alphabetically.
These data are real CLABSI data from 2011; however, the hospital was randomly pulled from the hospital list for demonstration purposes. Let’s review this hospital’s data and see how it compares to the predicted SIR of 1.0.
In Hospital X’s adult intensive care units during 2011, there were 5 CLABSIs identified and 2,611 central line days observed.
Based on the NHSN baseline data and the composition of locations in Hospital X, 5.3 CLABSIs were predicted.
This results in an SIR of 0.9 (O/P= 5/5.3), signifying that during this time period, Hospital X identified 10% fewer CLABSIs than predicted.
The 95% confidence interval (0.30, 2.21) indicates that the number of observed CLABSIs is not statistically different than the number of predicted CLABSIs. (Reminder: If 95% CI does crosses 1, the SIR is not statistically significantly different than 1.)

21. SIR comparisons
Hospital X’s SIR of 0.9 is higher than both the overall SIR for its bedsize category (0.6) and the SIR of all Virginia hospitals with adult ICUs (0.6).
Interpretation: Hospital X identified more CLABSIs than the average for its bedsize and in the state overall; however, it is not statistically significant.

22. New VDH CLABSI report: Hospital-specific graph shows annual CLABSI SIR over time
The CLABSI report will also include an annual SIR trend graph for each hospital, each bedsize category, and for the overall state. You will be able to view each graph by clicking on the state, bedsize category or hospital name from Table 1. As you can see, graphed are the annual SIRs for 2009, 2010, and 2011 for our sample facility, Hospital X.

23. Interpreting VDH CLABSI report graph
SIR point estimate
SIR 95% CI
Let us review step by step the components of the graph:
The same color legend is used throughout the report.
The horizontal dotted line is the reference line where the SIR is equal to 1. This is where the observed number of CLABSIs would be equal to the predicted number of CLABSIs.
The numbers are the SIR point estimates and the bars are the 95% confidence intervals for each year (calculated by NHSN). Again, the color of each circle summarizes the significance of the SIR point estimate and the 95% confidence interval.
If the 95% CI/the bars do not cross the dotted line, the SIR is statistically significantly different from 1. The circle will be red if it is significantly above 1 and green if it is significantly below 1. In 2009, we see that for Hospital X, the circle is red, meaning there were statistically significantly more CLABSIs observed than predicted that year.
If the 95% CI/the bars cross the dotted line, the SIR is not statistically significantly different from 1 and the circle will be blue. For Hospital X, this is true for 2010 and 2011.
SIR = 1
observed = predicted

24. New VDH CLABSI report: Table 2 has hospital-specific and comparison data
Table 2 corresponds with the graph and includes the data used to make the graph and comparison data for hospitals in that bedsize category. This table also includes the specific 95% confidence interval for each year; note this was not included in Table 1.

25. Central line-associated bloodstream infections (CLABSIs) in adult intensive care units standardized infection ratio by quarter, Virginia,
Trendline added using Excel to visually show overall increases or decreases
Line graph of SIR point estimates
This type of graph, which is not included in the VDH HAI CLABSI report, may be useful within facilities to help see beyond the usual fluctuations. The VDH HAI CLABSI report summarizes the SIR for an entire year; however, it may be helpful to see change over time during that year. For example, was there one quarter that was much higher or lower than the others? Was there a new practice implemented after which the trend seemed to go down more quickly?
The graph shows how the state CLABSI SIR has changed over time between 2009 and 2011 by plotting the statewide SIR for each quarter. We put the data in Excel to create a trendline. The trend demonstrates that the SIR has declined over time, even though there has not been a consistent decrease from quarter to quarter. The highest SIR occurred in the third quarter of 2009 and the lowest SIR occurred in the fourth quarter of Because the ultimate goal is sustaining zero CLABSIs, individual hospital infection prevention efforts should continue in addition to identifying and acting to address gaps. Since 2009, Virginia facilities have identified fewer CLABSIs than predicted in adult ICUs, reflecting already existing strong infection prevention efforts.
Since 2009, Virginia facilities have identified fewer CLABSIs than predicted in adult ICUs,
reflecting already existing strong infection prevention efforts.
Although there is not a consistent decrease from quarter to quarter, the overall trend
since 2009 has been decreasing.
Because the ultimate goal is sustaining zero CLABSIs, individual hospital effective
prevention efforts should continue in addition to identifying and acting to address gaps.

26. Resources
CDC's National Healthcare Safety Network (NHSN) HAI Summary Data Reports Q and A
html
NHSN e-News: SIRs Special Edition
VDH HAI website – surveillance

27. Dana.Burshell@vdh.virginia.gov 804-864-7550
Thank you!

28. Henrico Doctors’ Hospital IP Team Shared SIR with Leadership of Committee
Why you decided to educate your team
Hospital Compare website (1st Quarter Data)
VDH planning to use SIR
Corporate 2011 report using SIR (red, yellow, green)
Who you thought it was important to educate
Leadership in IP (Chairman, Chief Nursing personnel, Quality Director)
How did it go?
Explanation of SIR focused on > < 1 compared with NHSN national data
Example calculation
Examples of CLABSI corporate SIRs compared with CLABSI rates
Hospital Compare screenshot
Tables from VDH newsletter comparing CLABSI rates beside SIRs.
Nice display of confidence intervals.
Tips
Bring NHSN data summary reports to show where comparative data comes from
Next steps

29. Data from the February 2012 edition of the VDH HAI newsletter was used to show members of the committee how state CLABSI rate data compares with CLABSI SIR information annually for and quarterly for the most recent year (2011). -

30. Questions?