Adherence to Surgical Care Improvement Project Measures and the Association With Postoperative Infections FREE

The Surgical Care Improvement Project (SCIP), a national quality partnership dedicated to reducing the rate of surgical complications, has developed 20 measures covering various discrete elements of patient care.^1- 2 There are 9 publicly reported SCIP measures, 6 of which focus on postoperative infection prevention (Box). Adoption of these measures was supported by research attesting to their efficacy; and the development and implementation of these process-of-care measures have been endorsed by the National Quality Forum and other organizations that promote improvements in the quality of medical care.^1- 3

Individual SCIP Measures

INF-1: patients who received prophylactic antibiotics within 1 hour prior to surgical incision (2 hours if receiving vancomycin).

INF-2: patients who received prophylactic antibiotics recommended for their specific surgical procedure.

INF-3: patients whose prophylactic antibiotics were discontinued within 24 hours after surgery end time (48 hours for coronary artery bypass graft surgery or other cardiac surgery).

INF-4: cardiac surgery patients with controlled 6 AM postoperative blood glucose level (≤200 mg/dL [≤11.1 mmol/L]).

INF-6: surgery patients with appropriate surgical site hair removal with clippers or depilatory or those not requiring surgical site hair removal.

INF-7: colorectal surgery patients with immediate postoperative normothermia (first recorded temperature was ≥96.8°F within first 15 minutes after leaving the operating room).

Composite Measures

S-INF-Core: patient data on all 3 original Surgical Infection Prevention project perioperative infection-prevention measures (SCIP measures INF-1, INF-2, and INF-3).

S-INF: all patients with at least 2 recorded SCIP infection-prevention measures in a single visit (any combination of INF-1, INF-2, INF-3, INF-4, INF-6, and INF-7).

Hospital participation in these data collection efforts is voluntary. However, the Centers for Medicare & Medicaid Services (CMS) reduces hospital reimbursement by 2% if they fail to report their performance on these measures.^4- 7 After validation and cleanup of the data, the results are reported on the Hospital Compare Web site (http://www.hospitalcompare.hhs.gov).

Despite broad support from national stakeholders and a significant investment of time and money by the hospitals to collect these data, no large-scale investigation has been undertaken to evaluate their effectiveness for improving outcomes in routine clinical care. Single-institution studies attempting to evaluate the effectiveness of these measures have yielded mixed results.^8- 11 Therefore, the association between self-reported adherence to SCIP measures and reported complication rates is unknown, leading to debate over the necessity for participation in this large-scale data collection effort. This study evaluates the association between the 6 infection-prevention SCIP measures and postoperative infection rates in a representative sample of hospital discharges in the United States.

A retrospective cohort study was conducted using data from Premier Inc's Perspective Database¹² (an inpatient administrative database developed and maintained by Premier Inc; Charlotte, North Carolina), which contains data from the SCIP, to determine the effectiveness of self-reported adherence to 6 publicly reported SCIP infection-prevention measures in predicting postoperative infection. The study protocol and waiver of informed consent were reviewed and approved by the institutional review board of Case Western Reserve University (Cleveland, Ohio).

Premier Inc's Perspective Database, which is compliant with the Health Insurance Portability and Accountability Act,¹³ was used because it contains 100% of the discharge data from acute care hospitals (representing approximately 1 of every 5 discharges in the United States). Patient-level data undergo 95 separate quality assurance and data validation checks before they are made available for research. The SCIP data are from 398 hospitals.¹² The SCIP performance data have undergone all of the same quality assurance and data validation checks as the data reported on the Hospital Compare Web site. The data are the same discharge-level data submitted to the CMS for public report of hospital-level performance. Discharges meeting inclusion criteria for at least 1 SCIP quality measure between July 1, 2006, and March 31, 2008, were included, yielding 405 720 discharges from 398 hospitals and represent all regions of the United States.

This analysis makes use of a weighting variable developed by the CMS and Premier Inc for better representation of the US population. This weighting scheme is updated annually and projections match the approximately 35 million annual inpatient discharges with respect to age, sex, race/ethnicity, principle payment source, average length of stay, geographic region, hospital ownership, and total number of beds.¹² Patients with active infections were excluded from this analysis.⁴

The outcome measure is a dichotomous all-or-none variable developed for discharge-level analyses using International Classification of Diseases, Ninth Revision (ICD-9) codes. Complication codes indicated as being present on admission (the primary or the admitting diagnosis) were excluded. Postoperative infection indicates any discharge with a diagnosis of infection due to the operation (ICD-9 code 998.59). There are certain specific exclusions that are detailed in the ICD-9 code manual.¹⁴

In addition to the item-level analysis, the SCIP metrics were aggregated into 2 all-or-none measures of discharge-level adherence (Box). The S-INF-Core represents all patients who were eligible for all 3 original Surgical Infection Prevention project perioperative infection-prevention measures.^1,3 These measures deal with the timing of prophylactic antibiotic administration, appropriate prophylactic antibiotic choice, and discontinuation of prophylactic antibiotics within 24 hours. Discharges in which all 3 Surgical Infection Prevention process-of-care measures were collected and all 3 processes were appropriately received were assigned a value of 1. The S-INF represents all patients with at least 2 recorded SCIP infection-prevention measures. A discharge was assigned a value of 1 when the appropriate infection-prevention care was received every time it was indicated.

Data were available for patient age, sex, race, marital status, admission type, and insurance status. Patients were accounted for in the following age categories: 18 to 34 years, 35 to 54 years, 55 to 64 years, 65 to 74 years, 75 to 84 years, and 85 years or older. Race was grouped as white, black, and other. Admission type was documented by hospitals on the discharge forms as elective, urgent, or emergent. Health insurance information was aggregated into 5 categories: Medicare, Medicaid, commercial insurers, other governmental insurance, and uninsured. Other governmental insurance includes government programs such as corrections authorities, state agencies, Veterans Affairs funds, maternal and child funds, research funds, and any government-sponsored care other than Medicare and Medicaid programs.

Diagnosis and procedural information was available through ICD-9 codes. In addition to the all-payer–refined diagnosis-related group mortality risk provided by Premier Inc, a Charlson Comorbidity Index was calculated for each discharge to assess the effect of comorbidities using 2 independent methods. The Charlson Comorbidity Index was calculated based on the original description by Charlson et al,¹⁵ with the adaptation to an ICD-9 algorithm described by Deyo et al¹⁶ and Quan et al.¹⁷

Both the Charlson Comorbidity Index and the all-payer–refined diagnosis-related group mortality risk are shown in the descriptive analysis in Table 1; however, only the all-payer–refined diagnosis-related group mortality risk was used in the multivariable analysis because it explained a greater proportion of the variation than the Charlson Comorbidity Index. Furthermore, the all-payer–refined diagnosis-related group mortality risk is more familiar to hospital administrators because it is widely used within hospital databases.

Postoperative infection rates vary greatly by surgical procedure.^18- 20 Hundreds of procedures meet inclusion criteria for SCIP measurement; thus, procedural groupings must be implemented to allow procedural adjustment of analyses. All procedures relevant to a given SCIP measure are included in the analysis. Medicare uses procedural groupings to report procedural group–specific performance on SCIP measures to facilitate quality-improvement efforts; their procedural groupings²¹ were used to adjust for procedure in this analysis. The distribution of discharges by procedural groups are shown in Table 2.

Hospital size (small, medium, or large), setting (rural or urban), teaching status, and location (North, South, East, or West) were provided in the administrative extract. Other variables such as hospital and surgical volume were not provided because Premier Inc does not permit identification of participating hospitals.

χ² tests were used to compare adherent and nonadherent patient discharges with and without postoperative infections, available patient characteristics, operative characteristics, and hospital characteristics. Gynecological and urological procedural categories were combined with procedures not reported by Medicare in the multivariable analysis to account for the low number of postoperative infections in these groups. Hierarchical (multilevel) multivariable logistic regression analyses were used. These analyses take the correlation among observations within a cluster into account by using random-effect models to study the effect of discharge-level adherence to SCIP process-of-care measures with adjustments for the surgical procedure performed, patient characteristics, and hospital characteristics.

Unique models were developed to test the independent effect of each SCIP measure and each composite measure with postoperative infection using data from the entire timeframe studied. Once discharge-level associations were determined, tests for temporal trends were initiated. Temporal trends were tested by rerunning the discharge-level analyses on a quarterly basis. To ensure the stability of the measures over time, hospitals with fewer than 25 patients in the denominator for any 1 quarter were excluded from the temporal trend analysis. This is the same number of minimum patients used to ensure stability by the CMS prior to public report of hospital performance.²²

A detailed analytic plan was developed prior to data collection with an assumed infectious complication rate of 5%. To detect a 2% decrease in complication rates for adherent vs nonadherent patient groups with 80% power, 1605 patients in each group were needed using a 2-sided continuity-corrected χ² test with a significance level of .05. For purposes of maximizing generalizability of results to the US population, it was most appropriate to use the entire population available from Premier Inc in addition to the weighting estimates. The sample size used in each model exceeded these requirements. Data management and univariate analyses were performed using SAS software version 9.1 (SAS Institute Inc, Cary, North Carolina). Hierarchical nonlinear multivariable modeling was performed using HLM6 (Scientific Software International, Lincolnwood, Illinois). All tests were 2-sided and a P value of less than .05 was considered statistically significant.

Population characteristics are summarized in Table 1. Of the 405 720 discharges studied, 152 460 were men (37.6%), 140 964 were between the ages of 45 and 64 years (34.7%), 207 348 were married (51.1%), 185 432 were covered by Medicare (45.7%), and 278 535 were white (68.7%). Furthermore, 273 308 were elective cases (67.8%), 49 384 were operations performed urgently (12.3%), and 80 444 were performed emergently (20.0%). More than half (57.1%) of discharges identified were from medium-sized hospitals (100-300 beds) in the South (55.5%) (Table 2).

Patients with documented postoperative infections were more likely to be older, to have at least 1 comorbidity, and to have been admitted emergently. There were 3996 documented postoperative infections. All patient characteristics with the exception of race (P = .03) were statistically significant in their ability to predict infections in univariate analysis at the P value level of less than .001. Furthermore, all available hospital characteristics were statistically significantly associated with the likelihood of a postoperative infection prior to discharge. Unadjusted infection rates were highest in large, urban teaching hospitals in the Northeast.

Publicly reported, item-level adherence rates on the individual SCIP infection-prevention measures varied from 80% to 94%. A significant number of discharges had only 1 recorded SCIP measure (n = 120 316); the majority of these discharges (n = 119 456) recorded only SCIP INF-6, which measures appropriate hair removal. Seventy-eight percent of patient discharges reported adherence on all 3 prophylactic antibiotic measures when all 3 were reported (S-INF-Core); 73% of all discharges demonstrated adherence on all reported measures when at least 2 were reported (S-INF).

Demonstrated adherence to SCIP as measured through the all-or-none global composite S-INF was associated with a decreased likelihood of developing a postoperative infection from 14.2 to 6.8 postoperative infections per 1000 discharges (adjusted odds ratio [AOR], 0.85; 95% confidence interval [CI], 0.76-0.95) (Figure 1). However, the S-INF-Core composite was not statistically significant in its association with decreased likelihood of developing a postoperative infection (from 11.5 to 5.3 postoperative infections per 1000 discharges; AOR, 0.86; 95% CI, 0.74-1.01). In addition, reported adherence on individual SCIP items was not associated with decreased risk of postoperative infection. The measure for appropriate prophylactic antibiotic selection (INF-2) came the closest to approaching statistical significance with an AOR of 0.83 (95% CI, 0.69-1.00) and a decrease in infections from 21.0 to 7.5 postoperative infections per 1000 discharges.

There were 262 hospitals that had more than 25 patients with SCIP data in all 7 quarters. While all individual SCIP items and both SCIP composite measures demonstrated marked improvement in adherence rates by quarter, there was an increase in the rate of postoperative infections (Figure 2).

While the literature in support of each SCIP process-of-care measure is extensive, much of these data were based on early clinical trials yielding results on the efficacy of these actions, and not necessarily reflective of current actual practices or measure effectiveness.⁴ Furthermore, the few studies attempting to validate SCIP effectiveness have failed to show a positive relationship.^8- 11 Our analysis demonstrated no statistically significant association between reported SCIP adherence rates and postoperative infections as measured using individual SCIP process-of-care measures; however, development of all-or-none composite infection-prevention measures yielded statistically significant associations. Other studies of hospital process-of-care measures in nonsurgical areas have typically found weak statistical significance without definite clinical relevance.^23- 26

Our results are consistent with previous findings regarding public report of process-of-care quality data. Based on our findings, the individual item performance rates reported publicly do not fulfill their stated purpose of pointing consumers toward high-quality hospitals. However, when taken in aggregate, improved performance on our global all-or-none composite measure is associated with improved outcomes at the discharge level. Therefore, while the individual items may not imply quality differences, the overall ability to demonstrate adherence to multiple SCIP processes of care may. Improved methods for identification of quality of care are necessary to be able to define improvements in patient outcomes, and to justify the massive investment of time and money in tracking these processes of care.

While the efficacy of each process-of-care measure has been proven through clinical trials and is considered by many to be in line with clinical best practices, complication rates are likely influenced by many factors independent of these measures. Furthermore, demonstrated efficacy of a given practice is extremely different from demonstrated effectiveness of a nationally implemented process-of-care measure. Hence, non–SCIP-adherent care may not be inferior care. Best practices in prevention of complications dictates a multimodal approach; therefore, it is acknowledged that adherence to the SCIP measures is only part of effective patient care.^27- 28 Other aspects influencing surgical patient outcomes such as the skill and knowledge of the surgical team, a safe and clean working environment, and a general culture of quality surrounding patient care are likely to be at least as important to the assessment of hospital quality, and these aspects of a high-quality hospital may be better captured through aggregated measures.

Several authors^21,28- 29 have suggested that aggregate measures of quality are better indicators of high-quality care. It has been suggested that all-or-none measures of quality raise the bar,³⁰ and our data agree with this notion. Aggregating all-or-none measures to the individual patient level is perhaps a better approximation of the quality of care that the individual received. Ensuring that a patient received 4 of 4 quality practices is a more difficult task than demonstrating adherence on 1 measure, and the increased difficulty identified predictable differences in outcomes.

Despite this evidence, hospital-level performance on individual SCIP measures is publicly reported “to assist patients in selecting centers of excellence” for receipt of their surgical care.²² This publicized use implies that reported adherence on these measures is directly related to improved outcomes. Our findings are unable to support this assertion.

Furthermore, the collection and reporting efforts of these individual measures are a component of Medicare's value-based purchasing initiatives directed toward improving the quality of care in the United States through incentive-based reimbursement. Our inability to demonstrate a statistically significant association between individual process-of-care measures and clinical outcomes at the individual patient level suggests that the publicized rates, as reported on the Hospital Compare Web site, do not infer quality differences between hospitals. The lack of an associated quality difference further implies that implementing incentive-based reimbursement schemes on these individual items would do little to further improve hospital quality.

However, our findings do not necessarily indicate that the SCIP program was unsuccessful in improving quality. Improvements in quality may have come prior to the public reporting phase of SCIP implementation, and we would be unable to know if that were the case. Furthermore, the evidence that our aggregated measure of SCIP adherence was associated with decreases in postoperative infection rates does suggest that some association between SCIP adherence and quality exists.

Developers of the SCIP measurement process are hoping to demonstrate a 25% reduction in complication rates over 5 years¹; however, our findings are unable to suggest that the improvements in SCIP compliance have been associated with a reduction in infection rates. According to our estimates, increasing adherence to 100% would result in less than a 25% decrease in these rates.

There are several limitations to our study that should be noted. First, ICD-9 complication codes recorded on administrative discharge records have debatable sensitivity and positive predictive value.^31- 35 The magnitude of our associations may be minimized by the inaccuracy of our outcome measure. This issue biases our results toward the null, making it less valuable when our findings suggest we are unable to reject the null hypothesis. In addition, the absolute rates of infections noted in our report should not be construed as representative of actual postoperative infection rates because there is an assumed underreporting in administrative data sets. Furthermore, we were unable to account for infections that occur after discharge. Clinical data sets may be better suited to this purpose. It is important to note that because the collection of SCIP data through medical record review is completely separate from the process of recording complications in administrative records, the missed cases should be equally distributed between adherent and nonadherent groups by chance alone. However, we are unable to test this hypothesis. Finally, while further validation of this method is warranted, many authors have used ICD-9–based complication rates in publications in reputable medical journals, which demonstrate precedence for their use.^31,35- 36

Second, although use of the Premier Inc's Perspective data set allowed us to do a unique analysis at the patient level, the rates of adherence to quality measures may not be reflective of the rest of the country. Hospitals participating in the Premier Inc system may, on average, be more or less dedicated to improving the quality of their hospital. Therefore, the rates of adherence may be higher or lower in our sample population than seen on the Hospital Compare Web site, which represents all participating hospitals.²² We used patient-level weighting specifically developed by the CMS in conjunction with Premier Inc to permit national applicability of this data set to help minimize this effect.

Third, our trend analysis is imperfect because it lacks a control group. Therefore, our inability to demonstrate a decrease in complication rates may be due to underlying environmental changes and not the lack of association between SCIP measures and outcomes. This would be the case if complication rates were expected to increase at a rate greater than the rate we observed, and the efforts of the SCIP program and the increasing adherence prevented a more dramatic increase. There is no reason for us to expect that the complication rates studied should have increased at a more dramatic rate without hospital participation in SCIP. Following Medicare's announcement to financially incentivize hospital participation in quality reporting, more than 95% of all hospitals in the United States participate in these efforts.

Despite these limitations, our study demonstrated that although publicly reported adherence rates to SCIP process-of-care measures were associated with improved patient outcomes in our aggregated measures, the individual item relationships are weak and lack clinical significance. The improvements in SCIP adherence demonstrated over time were not associated with improved complication rates, and we were unable to demonstrate a clinically meaningful association between reported SCIP adherence and decrease in postoperative infections. Improved individual process-of-care measures and use of aggregation techniques in addition to improved data collection methods may be necessary to truly drive improvements in patient outcomes.