Safe but Sound: Title and subTitle BreakPatient Safety Meets Evidence-Based Medicine

The Institute of Medicine's seminal report To Err Is Human¹ highlighted the risks of medical care in the United States and shocked the sensibilities of many Americans. As one element of a multipronged response, the Agency for Healthcare Research and Quality (AHRQ) commissioned the University of California, San Francisco–Stanford University Evidence-Based Practice Center to develop a compendium of evidence-based patient safety practices, a resource summarizing the literature supporting practices relevant to improving patient safety.

Making Health Care Safer: A Critical Analysis of Patient Safety Practices² contains the complete results of this collaborative effort. Production of the report involved a commissioned group of 40 researchers across the country, including experts in patient safety, evidence-based medicine, and various areas of clinical medicine, nursing, and pharmacy. The report, which contains concise summaries of the evidence supporting more than 80 safety practices and a detailed description of its methods, has generated a substantial amount of attention (more than 50 000 copies have been ordered or downloaded) and some controversy. The latter, elegantly articulated in the accompanying article of this issue of THE JOURNAL by 3 of the dominant figures in the fields of patient safety and quality improvement,³ largely concerns the tension inherent in applying principles of evidence-based medicine to patient safety practices.

The paradigm of evidence-based medicine arose from the realization that health care interventions, no matter how commonsense or physiologically sound, often lack benefit and sometimes even cause harm.^{4 - 6} Since safety practices also may prove ineffective, wasteful, or even harmful, there is no reason to exempt most safety practices from the scrutiny of an evidence-based approach. Moreover, in the face of limited resources, evidence of effectiveness provides a useful parameter for prioritizing safety practices, just as with other health care interventions.

In the evidence report,² we defined a patient safety practice as a type of process or structure whose application reduces the probability of adverse events resulting from exposure to the health care system across a range of diseases and procedures. We intentionally avoided explicit reference to "error" in this definition, both because of its negative connotations and the difficulties in specifying what constitutes "medical error."⁷

The patient safety practices that flow from our definition (Table 1) may strike some readers as insufficiently distinct from quality improvement strategies. The same concerns have been raised regarding alternative definitions used by others.⁸ Nonetheless, our definition served several useful functions. First, the focus on prevention of adverse events due to medical care led to identification of a much broader pool of safety practices than would have been the case had we focused more narrowly on the prevention of errors. Second, physicians' judgments regarding the preventability of errors tend to be poorly reproducible.^{9 - 11} Third, our definition served a pragmatic function by allowing relatively unambiguous identification of practices that "reduce the probability of adverse events resulting from exposure to the health care system"—in other words, practices that make health care safer.

Using this definition, we identified 83 distinct safety practices supported by 70 systematic reviews and 293 additional primary investigations meeting our inclusion criteria,² a 10-fold higher yield than that of a recent systematic review¹² targeting interventions to reduce medical errors. Our definition allowed us to include the usual practices considered under the patient safety rubric, such as computerized physician order entry (CPOE) and strategies to prevent falls among hospitalized elderly patients, as well as practices that have not traditionally been grouped under patient safety (perhaps because they may not result from discrete "errors") but that clearly seek to prevent complications that fall within the same family of adverse events (Table 1).

To accommodate the heterogeneous practices that met our (or any) definition of patient safety, we modified existing frameworks for evaluating evidence to produce fairly liberal inclusion criteria.² In essence, we required that a patient safety practice must have had at least 1 study suggesting benefit. In addition, we insisted that such studies must have included a concurrent or historical control group (as in cross-sectional studies, before-after studies, or controlled clinical trials) and must have reported clinical outcomes (eg, adverse drug events [ADEs]) or intermediate outcomes plausibly related to patient outcomes (eg, serious medication errors).

Patient safety advocates have championed "root cause" analysis as a means of identifying factors that contribute to serious adverse events and of designing prevention strategies.^{13 - 15} It is important to recognize that, no matter how thorough or well performed, root cause analysis is essentially a hypothesis-generating activity, just as is its clinical counterpart, the case report. In high-stakes industries such as aviation and nuclear power, the low event rate generally does not permit epidemiologic investigation and subsequent hypothesis testing; hence, detailed accident investigations provide the foundation for many safety improvement efforts. In health care, however, many adverse events occur frequently enough to permit careful assessment of putative causative factors.

Consider the example of verbal orders. One can imagine an institution performing root cause analyses on a small number of medication errors and finding that verbal orders appeared to be important contributors to the errors. This commonsense conclusion might lead the institution to ban such orders. However, the only relevant empirical study¹⁶ reported a roughly 4-fold decreased risk of errors with verbal orders compared with handwritten ones.

This result does not imply that verbal orders are superior to handwritten ones, since they are used during unusual circumstances and likely undergo clarifications that handwritten orders may not (eg, a nurse confirms whether a confusing order is truly the desired one). Nonetheless, the single study of the impact of verbal orders on medication safety does not support the hypothesis that they are a major cause of medication errors. Therefore, pending additional data, strategies targeted at reducing verbal orders would be deemed to be non–evidence-based patient safety practices, analogous to the way unproven clinical practices would be characterized.

Some interventions seem so obviously beneficial that insisting on hypothesis testing will strike some as overly rigid. For example, the Institute for Safe Medication Practices recommends a number of prescribing strategies (eg, write "0.4 mg" rather than ".4 mg"),¹⁷ which, although unproven, are so sensible that they deserve widespread implementation. However, there are other "obviously beneficial" practices that do bear scrutiny. Perhaps the paradigmatic example of such a practice is the removal of intravenous potassium from general wards to prevent deaths due to iatrogenic hyperkalemia.^{18 - 19} A recent exchange on the National Patient Safety Foundation listserv highlighted the problems with even this apparently straightforward intervention. After concentrated potassium chloride was removed from the general floors of one hospital, ward personnel could not obtain potassium chloride solutions from the pharmacy quickly enough to meet their patients' needs, and some of the ward personnel began to hoard intravenous potassium chloride on their floors. Pharmacists were forced to chase after these hidden stashes, and intensive care units (which were allowed to continue to stock potassium chloride) quickly became de facto satellite pharmacies, informally distributing potassium chloride to ward personnel from other floors in an uncontrolled and chaotic fashion.²⁰ One listserv respondent wrote that this scenario demonstrated that even "well intentioned efforts to improve safety can misfire. [The case is also] a lesson about how flawed is the idea that advancing safety is a straightforward matter of removing hazards and forestalling human error."²¹ This example underscores the importance of applying evaluative standards to purported safety practices, even those that seem "obviously beneficial."

Concerns about the use of surrogate outcomes have long been raised in evidence-based medicine^{22 - 23} and have been justified by several landmark clinical studies.^{5 - 6} We believe these concerns also apply to patient safety. For example, fatigue among clinicians is commonly cited as an important patient safety target.¹ Sleep-deprived individuals function cognitively at the ninth percentile of the general population,²⁴ and studies²⁵ assessing the impact of fatigue on health care practitioners have documented increases in errors among sleep-deprived surgeons, anesthesiologists, and emergency physicians. Importantly, these studies involve focused assessments of errors during performance of specific cognitive tasks on simulators or in simulated scenarios.²⁶ The questions are to what extent these errors affect patients and whether fatigue (or its probable cause, long work hours) is so reliably linked to patient harm that it may be simply concluded that decreasing work hours or fatigue automatically implies that safety is enhanced.

We found only 2 studies^{27 - 28} that assessed the impact of fatigue on patient outcomes. A retrospective review²⁷ of 6371 operations and 351 associated postoperative complications found no increase in complications when the operating resident had already been on call for 24 hours. Although this study may be dismissed as inadequately designed to capture key outcomes, the second study²⁸ demands closer attention. This before-after analysis assessed the impact of New York State's restrictions on resident work hours following the Libby Zion case. The analysis revealed no change in hospital mortality with implementation of the new regulations, but also demonstrated an increase in complications and delayed test ordering.²⁸

Neither we nor the study's authors²⁸ would argue against addressing fatigue among health care professionals or dispute the successes of efforts to address fatigue among workers in industries where continuity is less important (eg, trucking, aviation). Nonetheless, this study highlights 2 important points. First, it demonstrates that changes in surrogate markers (in this case, fatigue or even errors due to fatigue) do not always match the effects on the true outcomes of interest (in this case, adverse patient outcomes). Second, changes to one aspect of a complicated system often produce unexpected collateral effects. In the New York study, even if work hour regulations reduced fatigue, they probably produced other effects (such as increased patient "hand-offs" and decreased continuity of care) that may have led to complications and delayed test ordering.

Leape et al³ contend that assessing the true outcomes of interest sometimes presents insurmountable technical or logistic difficulties, making studies of surrogate outcomes the only practical option. For example, they cite the major study of the impact of CPOE on ADEs, which showed a significant reduction in medication errors (the surrogate outcome) but a nonsignificant impact on ADEs.²⁹ Leape et al³ point out that the cost of this study exceeded $1 million. This cost, they imply, makes a study powered to detect a significant impact on the true outcomes of interest (ADEs) unfeasible. In our view, the cost in dollars (tens of billions) and person-hours (tens of millions) to implement CPOE at every US hospital would justify additional research to be certain that such systems make patient care safer. Of course, policymakers might well decide that the arguments supporting CPOE are so plausible or that other potential advantages, such as cost savings or improved provider education, are so compelling that it is not necessary to wait for additional studies. We have no fundamental disagreement with such policy choices as long as the presence or absence of supporting evidence is noted and considered during the deliberations.

The issue of generalizability commonly arises in considering whether interventions that appear efficacious in clinical studies will prove effective in routine clinical practice. Variations in technical skill, patient selection, and ancillary resources are just some of the factors that explain discrepancies between efficacy and effectiveness.³⁰ Again, the CPOE literature is illustrative.

Studies^{29 ,31} of the efficacy of CPOE have exclusively involved 2 internally developed ("homegrown") systems (a third, well-known internally developed system in Utah³² offers decision support for antibiotic ordering but not general order entry). Widespread dissemination of internally developed systems has generally been thwarted by idiosyncratic features of these systems and the challenges of linking them with legacy systems at other institutions. Although many vendors have produced commercial clinical information systems, most remain untested. Among the 344 clinical information systems approved by the Joint Commission on the Accreditation of Healthcare Organizations in 1998, 100 systems had not had even a single user.³³ Thus, the efficacy of CPOE systems available to most hospitals remains unknown.

Generalizing the results of internally developed, local CPOE systems also may be compromised by implementation problems. Even when failure is not so extreme as the shutting down of the system (as happened in one early effort³⁴ ), many institutions report poor compliance with their fully functioning CPOE systems. In one survey,³⁵ 57.7% of hospitals with fully implemented CPOE found that more than 90% of orders continued to be handwritten, bypassing the implemented CPOE system.

In addition to being an important principle of evidence-based medicine, considering the potential harm from any practice is consistent with the experience drawn from efforts to improve safety outside health care. In fact, one of the major lessons of the safety literature is that this year's safety measure often contributes to next year's incident, because of specific unintended effects or the increased complexity of the new system.³⁶ Moreover, safer technologies sometimes produce "compensatory" increases in risk—for example, safer cars that people drive more recklessly and protective sports gear that promote more damaging player contact.³⁷

Accordingly, when we evaluated and ranked safety practices in Making Health Care Safer,² we considered the "need for vigilance." By this we meant that certain practices, although not harmful per se, might lead to changes in clinician behavior or have other unintended consequences that could undermine their benefit, such as in the previously cited example of the regulation of resident work hours in New York State.²⁸ Similarly, CPOE systems, even though they prevent many errors, may have design flaws that generate specific hazards and require vigilance to detect. One CPOE study demonstrated an initial increase in intercepted potential ADEs attributed to the design of the ordering screen for potassium.³⁸ Although this error was promptly rectified, it underscores the importance of ongoing vigilance for unintended harmful effects from complex interventions, even those considered "safety practices."

Although not a specific principle of evidence-based medicine, we considered in the report² the degree to which individual practices were already being used. This consideration reflected our charge from AHRQ to highlight practices that decreased important patient safety problems and that were not already being widely implemented. The paradigmatic example here was that of double read-backs before blood transfusion, a practice that has undoubtedly prevented transfusion errors and is nearly universally used. We regarded promotion of such practices as less likely to affect patient safety than other effective but less widely implemented practices and therefore structured our rating scheme to adjust downward the rating of practices that were already widely used. This implementation factor affected the ratings of only 4 (5%) of 83 practices, and even these practices were downgraded only slightly based on this consideration.

Leape et al³ are particularly concerned about the downgrading of unit dosing, the practice of dispensing medications in ready-to-administer packages, vs the traditional system of preparing doses from medication stock kept in patient care areas. Although unit dose systems are accepted as standard practice in the United States, a study³⁹ comparing a US hospital using unit dosing with a British hospital using the traditional ward stock system found the British hospital to have less than half (3.0% vs 6.9%) the errors. This study has some important methodological limitations, but the 4 observational studies indicating benefit from unit dose systems generally share the same limitations.⁴⁰ Moreover, all the studies involved errors and not ADEs. Thus, the relatively low rating assigned to unit dose drug distribution (fourth of 5 categories for overall evidence² ) largely reflected the relatively weak evidence of effectiveness rather than downgrading due to current implementation.

In responding to the concerns raised by Leape at al³ and defending an evidence-based approach to patient safety, we have discussed practices (decreasing verbal orders, work hour limitations, and removing potassium chloride from ward stock) that, although intuitively attractive, generally lack supporting evidence. Our charge in producing Making Health Care Safer² was to identify evidence-based patient safety practices, and thus the report describes many other practices that do have substantial supporting evidence. Leape et al³ do not dispute the central premise that implementing such evidence-based practices will improve patient safety. Rather, their principal concern is that (unrealistic) insistence on high-quality evidence will prevent the adoption in health care of practices that appear to have been effective in other, "safer," industries.

Although we share the enthusiasm of Leape et al about the potential benefits of such practices,^{41 - 43} we also wonder just how "the value of most safety measures has been established directly or by analogous practices in other industries" in the absence of evidence. Such non–evidence-based practices not only can cause harm, but their implementation can carry opportunity costs as well. In a world of limited resources, the choice to implement a non–evidence-based practice often represents an implicit choice not to provide adequate nurse staffing of intensive care units, subsidize activities of clinical pharmacists, or pay for practices that prevent complications among hospitalized elderly patients.

Leape et al³ argue that applying the framework of evidence-based medicine will create limitations and constraints that ultimately prevent health care from emulating the safety records of other high-risk industries. However, the problem in health care has not been a lack of tools to improve patient safety, but the relatively low priority this goal has traditionally received. The last several years have been a hopeful beginning in this regard, but patient safety research and practice remain in their infancy.

In the end, we agree with Leape et al³ that the best approach for ensuring patient safety will be one in which the general insistence on evidence does not prevent implementation of practical, low-risk, but understudied interventions that seem likely to work. We highlighted this balanced approach in our report² when we wrote,

Healthcare clearly has much to learn from other industries. Just as physicians must learn the "basic sciences" of immunology and molecular biology, providers and leaders interested in making healthcare safer must learn the "basic sciences" of organizational theory and human factors engineering. Moreover, the "cases" presented on rounds should, in addition to classical clinical descriptions, also include the tragedy of the Challenger and the successes of Motorola. On the other hand, an unquestioning embrace of dozens of promising practices from other fields is likely to be wasteful, distracting, and potentially dangerous. We are drawn to a dictum from the Cold War era—"Trust, but verify."