Surgical Care Improvement: Title and subTitle BreakShould Performance Measures Have Performance Measures

The Surgical Care Improvement Project (SCIP) was established in 2006 with the goal of reducing surgical complications by 25% in 2010.¹ Of the 9 performance measures, 6 are related to surgical site infection prevention.¹ Efforts to reduce surgical site infection are important because this complication results in significant morbidity and additional resource use. To this end, the SCIP was designed to improve adherence for prophylactic antibiotic administration, as well as processes related to glucose control, hair removal from the surgical site, and intraoperative normothermia, in patients undergoing elective surgical procedures. It has achieved this goal to the extent that hospitals have successfully implemented these processes. However, what evidence exists demonstrating that improved adherence has achieved the goal of reducing surgical complications?

Complications from surgical procedures are costly and cause substantial patient morbidity. Patients need transparent information guiding them to the best hospitals for their surgical procedure and third-party payers need a mechanism to influence and make accountable the care of patients undergoing expensive procedures. The time, resources, and importance placed on SCIP process measures lead to the conclusion that adherence to these measures can discriminate practices associated with optimal surgical outcomes. The SCIP measures were added to the Hospital Compare Web site (http://www.hospitalcompare.hhs.gov) with the goal of guiding patients about where to receive surgical care, based on the underlying assumption that higher performance on these measures equals better surgical outcomes. Furthermore, the Centers for Medicare & Medicaid Services has proposed use of the SCIP measures for value-based purchasing and payment to hospitals.²

The report by Stulberg et al³ in this issue of JAMA is the largest study to date that fails to demonstrate an association between adherence to SCIP process measures and the occurrence of postoperative infections. The authors found no significant association between individual process measures or the all-or-none composite core measurement composed of all 3 measures for prophylactic antibiotic administration and postoperative infection. They report a modest association between adherence to a composite measure that included at least 2 of the 6 SCIP measures applied to an expanded SCIP population and postoperative infection. Despite substantial improvements in SCIP adherence over the 2-year study period, postoperative infection rates actually increased.

The SCIP includes 3 core infection-prevention measures: SCIP-1, antibiotic administration within 60 minutes prior to incision; SCIP-2, appropriate selection of antibiotic coverage; and SCIP-3, timely discontinuation of antibiotics. The SCIP-2 measure ensures that adequate antimicrobial coverage is administered with a focus on limiting too broad of coverage. Similarly, for the SCIP-3 measure, data support that prolonged coverage has no incremental improvement in surgical site infection rates and is potentially harmful due to the development of resistant organisms. Of these 3 measures, there has been evidence to support that the SCIP-1 measure reduces the incidence of surgical site infection, although that evidence involves whether the prophylactic antibiotic is administered at all, and not the timing of antibiotic administration. Other studies have shown that even though administration of antibiotics in the operating room resulted in better SCIP-1 adherence,^{4 - 5} there was no association with lower surgical site infection in patient- or hospital-level analyses.^{5 - 12}

Delivery of preoperative prophylactic antibiotics improves surgical site infection rates in the appropriate clinical setting¹³ and all patients in the study by Stulberg et al³ received prophylactic antibiotics. However, the SCIP-1 measures a dichotomous variable based on a continuous event (the time between antibiotic administration and surgical incision). This measure presumes that any antibiotic administered within 60 minutes prior to incision is effective, but that antibiotics administered outside that interval are not. To date, no prospective trials have validated the 60-minute window as a measure to discriminate meaningful differences in patient outcomes. Between 2002 and 2009, at least 8 reports that evaluated the association between SCIP-1 adherence and surgical site infection among a combined 31 448 patients undergoing major surgical procedures in 301 hospitals showed little, if any, evidence of increased surgical site infection rates for patients receiving antibiotics more than 60 minutes before the incision is made compared with patients receiving antibiotics within 60 minutes prior to skin incision.^{5 - 12} In a study based on national Veterans Affairs data, 75% of timing errors were due to administration of antibiotics more than 60 minutes before incision time.⁴

In-depth analysis of these studies only muddies the waters. The SCIP-1 measure is defined as initiation of antibiotic (not completion of the infusion). In 3 studies, antibiotic administration initiated within 15 to 30 minutes prior to incision was associated with higher surgical site infection rates than antibiotic administration 30 to 60 minutes before surgical incision, suggesting that drug administration too close to the time of incision, while meeting SCIP-1 adherence, may not be an effective practice.^{12 ,14 - 15} These studies also raise concerns that in some settings, pushing the timing of antibiotic administration close to incision time to achieve adherence with the SCIP measures may result in less effective prophylaxis, perhaps explaining to some extent why adherence to the SCIP-1 measure has not been shown to be associated with lower rates of infection. It appears that even a metric as simple as timely administration of antibiotics is complicated by multiple patient, pharmacological, and procedural factors. In the pursuit to create metrics that can be measured, practices that are less effective than those targeted for improvement may have been created.

Taken together, the available evidence generated by post-SCIP implementation analyses suggests that the window of effectiveness for antibiotic administration is dependent on patient, procedure, and antibiotic variables and does not fit neatly within a 60-minute time frame. Moreover, the current metrics do not appear to discriminate between effective and noneffective care at the patient or hospital level. Perhaps all the improvement in surgical site infection related to prophylactic antibiotic administration has been achieved. Considering that adherence rates for prophylactic preoperative antibiotic administration, on average, have surpassed 90%,³ and given the shrinking budgets of most hospitals, efforts should be limited to metrics that have the most potential to further improve outcomes and are tightly linked to the desired outcomes. From the report by Stulberg et al,³ and the currently available literature, it appears that investing resources in SCIP reporting is no longer cost-effective.

Ideal surgical quality-improvement efforts would measure whether the right patient receives the right operation at the right time and whether the operation is effective. This is more challenging to measure given the complex nature of the health care system, including barriers to access, financial incentives, and the uniqueness of the therapeutic intent of individual surgical procedures. Current mandated surgical quality-improvement processes such as SCIP focus on incremental and narrow process measures that are purported to measure the overall quality of an episode of surgical care. Despite enormous resources committed to these measures and marked improvement in adherence, the evidence to date suggests that SCIP has not improved surgical outcomes. Future quality-improvement endeavors should have linkage between discrete performance and outcome measures so the effectiveness of combined efforts can be unequivocally measured and clearly evaluated.