Quality of Traditional Surveillance for Public Reporting of Nosocomial Bloodstream Infection Rates FREE

Public reporting of hospital-specific infection rates is widely promoted as a means to improve patient safety.^1- 2 Central line–associated bloodstream infection (BSI) rates are considered a key patient safety measure because such infections are frequent,³ lead to poor patient outcomes,⁴ are costly to the medical system,⁵ and are preventable.^6- 7 Publishing infection rates on hospital report cards, which is increasingly required by regulatory agencies, is intended to facilitate interhospital comparisons that inform health care consumers and provide incentive for hospitals to prevent infections.⁸ Interhospital comparisons of infection rates, however, are valid only if the methods of surveillance are uniform and reliable across institutions.^9- 10

Most hospitals performing central line–associated BSI surveillance rely on infection preventionists (formerly known as infection control practitioners¹¹) to manually perform central line–associated BSI surveillance. The infection preventionists apply surveillance case definitions published by the Centers for Disease Control and Prevention (CDC).¹² These case definitions rely on a mixture of objective criteria (positive blood cultures) and subjective criteria (judging whether bacteremia is from a central line rather than secondary from an extravascular source, or determining whether recovery of a common skin commensal in the blood represents a true infection vs contamination).

In practice, surveillance is challenging because it is time consuming and lacks a gold standard for validation. Case finding by infection preventionists is effort dependent and can lack sensitivity, resulting in underreporting of infection rates.¹³ Furthermore, subjective components of surveillance criteria may be inconsistently applied by infection preventionists, reducing the validity of interhospital and potentially intrahospital comparisons.¹⁴ In spite of these threats to validity, traditional central line–associated BSI surveillance by infection preventionists has been rarely validated against external measures.^13,15

The recent development of computer algorithms for central line–associated BSI surveillance provides an opportunity to establish an objective reference standard with which to benchmark infection preventionist determination of infection rates. A computer algorithm can be applied consistently among different institutions, using only objective criteria (microbiologic, pharmacy, and patient location records) in an identical and comprehensive manner.^16- 17 We used a computer algorithm that had been developed to approximate the prevailing CDC surveillance definition¹² for central line–associated BSI surveillance from 1988 through the study period¹⁸; the algorithm was previously found to be as accurate as infection preventionist determinations when compared with an expert reference standard in a single institution.¹⁹

As a part of a research collaborative supported by the CDC Prevention Epicenter Program, we performed a study involving 4 medical centers, comparing ecologic (unit-period) central line–associated BSI rates determined by traditional infection preventionist surveillance with rates determined by a computer algorithm reference standard. Our study aims were to estimate the overall correlation between infection preventionist and computer algorithm ecologic rates, and to test for institution-dependent heterogeneity in this relationship. We also assessed whether institution-dependent variability, if found, would lead to relevant differences in the relative ranking of institutions based on reported central line–associated BSI rates.

Four academic medical centers (2 in Chicago, Illinois; 1 in Columbus, Ohio; 1 in St Louis, Missouri) participated in this study. A convenience sample of 20 intensive care units (ICUs) across the 4 medical centers contributed electronic data and infection preventionist surveillance data that had been collected from 2004 to 2007. For analysis, infection rates for each ICU were aggregated into 12-month periods. These blocks could be nonconsecutive to exclude months when surveillance was not carried out by infection preventionists. Individual ICUs were asked to contribute 2 to 3 consecutive unit periods of data for the study, based on data availability.

All central line–associated BSI rates were expressed as (number of central line–associated BSI events)/(1000 central line–days). Central line–days were obtained from daily counts provided by each ICU's nursing unit using CDC methods²⁰; these counts were used as denominators to calculate rates for both infection preventionist and computer algorithm measures.

This study was powered to test the null hypothesis that there is no correlation between computer algorithm and infection preventionist central line–associated BSI rates. Using the parameters of an effect size (r) = 0.8 (based on prior estimation¹⁹), 2-sided α = .05, and β = .10, a sample size of 34 total unit-periods would be required.²¹

Each medical center obtained approval from its respective institutional review board for human subjects research; informed consent was waived, and 1 medical center excluded prisoners from review.

As a part of routine infection control activity at each medical center, ICU-specific central line–associated BSI rates were prospectively measured by infection preventionists using CDC surveillance definitions applicable during the study period. All infection preventionists employed at each medical center were registered nurses, medical technologists, or microbiologists trained in infection control, and all were blinded to their participation in this study.

Each medical center's infection control department worked independently, with unique organizational and informatics approaches for case finding and information gathering. Although each medical center used the hospital electronic medical record to generate lists of positive blood cultures for case finding, infection preventionists applied CDC central line–associated BSI definitions manually, without the assistance of computer-decision support.

We used a computer algorithm that approximated the CDC surveillance definition used by infection preventionists during the study period. The algorithm (code available at http://bsi.cchil.org) retrospectively calculated central line–associated BSI rates within ICUs at each medical center, using clinical data in the electronic medical record. The algorithm evaluated laboratory-identified blood culture isolates in an objective fashion, using stepwise logic to make a central line–associated BSI determination (Figure 1); further details are published elsewhere.¹⁸

Common skin commensal organisms (defined as diphtheroids, Bacillus, Propionibacterium, coagulase-negative Staphylococcus, or Micrococcus species) recovered from blood cultures can represent either infection or blood culture contamination. The computer algorithm considered either the recovery of at least 2 positive common skin commensal cultures of the same species within 2 hospital days or a single positive common skin commensal culture with vancomycin administered within the 2 subsequent days as representing an infection.

Computer algorithms were distributed to each participating center in both logical representation and SQL (Structured Query Language) code.¹⁸ Each medical center adapted the code to its clinical database, and correct implementation was ascertained by verifying results from a standardized test data set.

Central line presence was determined by manual chart review at 3 medical centers; the fourth center was able to automate central line detection. Only blood culture episodes determined to be ICU-related and primary by the computer algorithm were manually reviewed. Central line data were abstracted from the nursing daily flow sheet; a central line was considered present if documented on the day of a positive blood culture result or up to 2 hospital days prior. Nursing flow sheets were available for all patients by chart review with complete ascertainment of central line status.

We compared 12-month unit-period infection rates from infection preventionist and computer algorithm surveillance using several methods. Differences between the rates defined by the infection preventionist and computer algorithm were tested using the Wilcoxon signed-rank test, and strength of correlation was tested using the Spearman rank-order correlation. Next, we created linear regression models of infection preventionist rates vs the computer algorithm rates, stratified by medical center. In a final linear regression model with all medical centers combined, we tested for heterogeneity of slope and intercept using interaction terms with medical center as a nominal variable. All regression analyses were weighted to account for differences in the magnitude of the denominator (central line–days) among unit periods. The 95% confidence intervals (CIs) for central line–associated BSI rates were calculated assuming binomial distribution.

All analyses were performed using SAS version 9.1.3 (SAS Institute Inc, Cary, North Carolina) and Stata version 9.0 (StataCorp, College Station, Texas). P < .05 was considered statistically significant.

Twenty ICUs in 4 medical centers contributed 41 twelve-month unit periods, representing 241 518 patient-days and 165 963 central line–days and included 9 types of units: 5 medical, 4 surgical, 1 combined medical-surgical, 2 neurosurgical, 3 cardiac, 1 oncologic, 1 cardiothoracic surgery, 1 burn, 1 bone marrow transplant, and 1 trauma.

Across all unit periods, the median infection preventionist–measured central line–associated BSI rate was 3.3 infections per 1000 central line–days (interquartile range [IQR], 2.0-4.5; range, 0.4-8.5). The median rate determined by the computer algorithm was 9.0 per 1000 central line–days (IQR, 6.3-11.3; range, 2.0-21.5). The median rates for the infection preventionist and computer algorithm methods were significantly different (P < .001).

We calculated the Spearman rank correlation coefficients to determine how well the infection preventionist rates correlated with computer algorithm rates on a unit-period basis. When unit periods were analyzed in aggregate across medical centers, overall correlation (Spearman ρ) between computer algorithm and infection preventionist rates was weak, 0.34 (95% CI, 0.04-0.59; P = .03). However, when stratified by medical center, we found that the point estimates of the correlations varied widely (medical center A: ρ, 0.83; 95% CI, 0.05 to 0.98; P = .04; medical center B: ρ, 0.76; 95% CI, 0.32 to 0.93; P = .003; medical center C: ρ, 0.50; 95% CI, −0.11 to 0.83; P = .10; medical center D: ρ, 0.10; 95% CI, −0.53 to 0.66; P = .77).

In the linear regression models for each medical center (Figure 2), the computer algorithm rates were generally higher than the paired rate defined by the infection preventionist, and the slopes of the fitted linear regression lines were less than 1 (range, 0.28-0.63). The goodness-of-fit or R², representing how closely the observations clustered around the regression line, varied widely between hospitals (Figure 2). The R² values for each of the medical center regression lines were as follows: medical center A, 0.43; medical center B, 0.81; medical center C, 0.36; medical center D, 0.28. Higher R² values represent more consistency between infection preventionist and computer algorithm determinations across unit-periods.

When we included all data into a single linear regression model, we found that although the slope did not differ significantly by medical center (P = .67), the intercept was significantly different across medical centers (P < .001). Because no differences in slope were detected, our final model (Figure 3) represented the study data with 4 parallel lines with equal slope (0.43) and different intercepts (range, −2.99 to 1.47) for each medical center. Thus, any given single computer algorithm rate would correspond to a different predicted infection preventionist surveillance rate at each institution (eg, for a hypothetical computer algorithm rate of 9 central line–associated BSIs per 1000 central line–days, the predicted infection preventionist rate would vary by hospital, from 1.1 infections for medical center C to 4.9 for medical center A).

Relative rankings of the 4 medical centers differed depending on the surveillance method (Figure 4). Under infection preventionist surveillance, medical center C reported the lowest central line–associated BSI rate (pairwise comparison with the next lowest medical center D not statistically significant, P = .07). When reanalyzed with the computer algorithm, medical center C had the highest central line–associated BSI rate, and rates were statistically different across all 4 medical centers (P < .01).

We compared central line–associated BSI rates from infection preventionist surveillance with a computer algorithm reference standard in ICUs across 4 medical centers. We hypothesized that although we would find differences in the absolute infection rates estimated by infection preventionist surveillance as opposed to computer algorithm surveillance, we would find reasonable and uniform ecologic correlation between the 2 methods. Instead, we found weak overall correlation between the 2 methods and importantly, variable correlation when stratified by medical center.

Medical center–specific variation in infection preventionist surveillance was confirmed through linear regression modeling; we found that for a given computer algorithm central line–associated BSI rate, the expected rate defined by infection preventionist surveillance varied significantly by medical center. The center-specific variation markedly affected the rank order of institutions, such that the medical center with the lowest rate as reported by infection preventionists had the highest rate by the computer algorithm reference standard. Our findings suggest that there is local variation in central line–associated BSI surveillance performance at different medical centers, raising concern for the validity of interinstitutional rate comparisons.

In the past several years, public reporting of hospital infection rates has been increasingly promoted as a means of comparing patient safety among different institutions. In the United States, roughly one-quarter of all states have legislative mandates to report central line–associated BSI rates from ICUs.²² These data are used by patients, advocacy groups, and regulators to compare hospitals.²² Furthermore, hospital reimbursement is increasingly dependent on reported rates,²³ creating potential financial incentives for hospitals to underreport rates. Our study highlights the potential fallibility of traditional surveillance methods using partially subjective criteria and underscores the need for cautious interpretation of these results until more reliable measures or validation against objective measures can routinely be performed.

Prior studies have analyzed the validity of infection preventionist central line–associated BSI surveillance through retrospective case review, although none to our knowledge has focused on variation in surveillance practice across medical centers. A pilot validation of the CDC surveillance definitions was performed in 1991 through1993 at 9 hospitals, with 1 ICU per hospital contributing a stratified random sample of 15 charts with central line–associated BSIs, as well as additional charts without them, for standardized review by trained infection preventionists and CDC epidemiologists.¹⁵ The study found that infection preventionist surveillance was 85% sensitive and 98% specific compared with the expert review. Although these favorable performance characteristics formed the evidence base for future surveillance, the study was performed before the onset of widespread public reporting and was not designed to look for differences in infection preventionist performance across medical centers.

More recently, poor sensitivity of surveillance conducted by infection preventionists was found during a retrospective review of a sample of blood cultures with potential central line–associated BSI in 6 Australian hospitals using a CDC-adapted surveillance system with the same criteria used in the present study. Compared with expert review, infection preventionist surveillance was found to have poor sensitivity (35%) and only marginal agreement (κ = 0.31).¹³

For interinstitutional comparison of infection rates, consistent surveillance (ie, reliability) among institutions may be even more important than accuracy. In a mathematical simulation comparing a purely objective approach (analogous to the computer algorithm used in the present study) with a partially subjective approach (performed by infection preventionists using clinical criteria) to rank institutions by central line–associated BSI rates, the objective method overestimated the true rate but was more accurate in ranking institutions by their rates.²⁴ Thus, removing subjectivity (ie, clinical judgment) potentially leads to reduced accuracy but improved reliability, leading to more accurate rankings of institution-specific rates.

Discrepancies in surveillance practice may vary substantially across 2 domains, case finding and classification. Case finding of potential BSIs can be incomplete. For example, although a patient may have multiple blood cultures postive for central line–associated BSI throughout a prolonged ICU stay, not all blood cultures may be investigated, especially if the infection preventionist does not have a systematic method of tracking blood culture results. For classification, there is likely substantial variability in application of subjective aspects in the central line–associated BSI definition. For example, infection preventionists (and clinicians) will not always agree whether a positive blood culture originated from a central line or from an extravascular source such as an intra-abdominal abscess as a secondary BSI. In instances in which definitive information is not available, infection preventionists may differentially classify ambiguous cases.

To further explain the observed variability in rates, qualitative differences may exist at the institution level, such as local clinical culturing practice, quality of medical documentation, and strength of institutional oversight over infection prevention activities. Further qualitative assessments may be warranted to better understand the relative importance of institutional and episode-level sources of variability. All 4 medical centers are academic training institutions with strong interest in infection prevention and research; thus, we speculate that although institutional differences may exist, most of the variability observed in this study is explained by differences in infection preventionist performance of surveillance.

The use of computer algorithms for central line–associated BSI surveillance has been described in prior studies.^19,25- 26 The computer algorithm surveillance used in the current study was previously validated in a single-center study and was found to have substantial agreement with expert review (κ = 0.73).¹⁹ Distribution of the same computer algorithm to the 4 medical centers in the present study permitted the efficient estimation of an objective reference rate across a large number of ICUs and unit-periods, allowing for detection of clinically and statistically significant differences in infection preventionist surveillance performance across institutions. The computer algorithm has inherent limitations; in particular, it performs secondary infection determinations using information from nonblood cultures, which may not always be accurate. Furthermore, like the infection preventionist, it may be affected by the local propensity of hospitals to obtain clinical cultures. Future studies that analyze agreement at an episode level, with an additional reference standard, are needed to validate the computer algorithm for generalized use.

Our study has several limitations. First, although we use the computer algorithm as a reference standard, the true central line–associated BSI rate is unknown (and outside of simulation models, likely unknowable). We selected the computer algorithm as a reference standard because it could be consistently applied across institutions and because of its prior validation against expert review.¹⁹ To ensure uniform application, we tested each medical center's installation of the computer algorithm by providing standardized input data sets to ensure that local installations of code produced identical results.¹⁸ Second, we studied agreement between ecologic rates at the unit-period level, not at the individual blood culture episode level. Even if there is excellent agreement at the ecologic level, there may be considerable disagreement on the significance of individual blood culture episode results. For the purpose of this study, because routine reporting and public ranking occurs at an ecologic level, we focused on the ICU as the unit of analysis rather than the individual patient. Third, although we used the CDC surveillance definitions contemporaneous to the study period, the CDC changed a component of the surveillance definitions in January 2008; the criterion that allowed a single positive blood culture caused by common skin commensals to represent an infection if appropriate antibiotic treatment was administered was removed.²⁷ Whether the use of the new surveillance definition improves the correlation between computer algorithms and infection preventionist rates requires further study because a large subjective component of infection preventionist review (determination of primary vs secondary infection) remains.

Central line–associated BSI surveillance, whether performed by infection preventionists or computer algorithms, is not designed to guide or replace clinical diagnosis by physicians; rather, it is a specialized tool to provide consistency in monitoring rates over time and among institutions to inform infection prevention efforts. In this study, we found strong evidence of institutional variation in central line–associated BSI surveillance performance among medical centers. Inconsistent surveillance practice can have a significant effect on the relative ranking of hospitals, which threatens the validity of the metric used by both funding agencies and the public to compare hospitals. As central line–associated BSI rates gain visibility and importance—in the form of public report cards, infection reduction campaigns such as “Getting to Zero,”²⁸ and financial incentives for reducing rates by private insurers and the Centers for Medicare & Medicaid Services²³—we should seek and test surveillance measures that are as reliable and objective as possible.