What Practices Will Most Improve Safety?: Title and subTitle BreakEvidence-Based Medicine Meets Patient Safety

The Institute of Medicine (IOM) report To Err Is Human¹ converted an issue of growing professional awareness to one of substantial public concern in a manner and pace unprecedented in modern experience with matters of health care quality. The epidemiologic finding that more than 1 million injuries and nearly 100 000 deaths occur in the United States annually as a result of mistakes in medical care came from studies nearly a decade old, but it was new information for the public, and it resonated strongly. In short order, the US Congress initiated hearings and the president ordered a government-wide feasibility study, which led to a subsequent directive to governmental agencies to implement the recommendations of the IOM report. The IOM called on all parties to make improving patient safety a national priority. In response, physicians, hospitals, and health care organizations have been searching for safe practices and asking what they should do to make health care safer.

Anticipating this need, the IOM report also recommended that the Agency for Healthcare Research and Quality (AHRQ) determine which safety practices are effective and disseminate a list of "best practices" to all clinicians. Responding to this appeal, AHRQ requested the National Quality Forum to use a consensus process of experts to define a list of best practices. To inform this process, it also commissioned the Evidence-Based Practice Center (EPC), University of California, San Francisco–Stanford University, to evaluate the evidence supporting a long list of proposed safety practices. Given a 6-month time frame, the EPC enlisted numerous experts nationwide to conduct the analyses. The resulting report by Shojania and colleagues² is a superb and ground-breaking compendium of what is known about the evidence of effectiveness for certain topics of interest in the science of preventing complications. Where practical, these practices should be implemented.

Evidence-based assessment, as conducted by the EPC, is a formal method of literature analysis that uses standardized techniques and places heavy emphasis on data from randomized controlled trials.^{3 - 4} Advocates of evidence-based medicine (EBM) argue that medical decisions should be based, as much as possible, on a firm foundation of high-grade scientific evidence, rather than on experience or opinion. The motivation for EBM stems from the observation that many widely used practices lack supporting evidence and are therefore of questionable value. In the past, many experience-based and opinion-based practices have proved to be ineffective or even harmful. For patient safety practices, to our knowledge, no thorough, evidence-based assessment of the literature had been performed previously.

Thus, the publication of Evidence Report 43 in July 2001 was eagerly anticipated by health care leaders who are anxious to improve safety but not sure what to do.² However, to their surprise and confusion, many found that a number of the improvements in safety practice they have been working so hard to implement are not even mentioned in the report (Table 1a). Furthermore, of the 11 practices Shojania et al² recommend most highly as "clear opportunities for safety improvement" because they met the formal criteria for strength of evidence regarding impact and effectiveness, only 3 have been usually characterized as safety issues at all: anticoagulation for prevention of deep venous thrombosis, antibiotic prophylaxis to prevent surgical infections, and use of pressure-relieving materials to prevent pressure ulcers. Conspicuously absent were many simple and well-accepted changes, such as the 15 best medication practices endorsed and recommended by the Massachusetts Hospital Association and the American Hospital Association.^{6 - 7} Only 3 of these 15 (unit dosing, computerized physician order entry [CPOE], and bar coding) were discussed in the report, and none were placed in the top category because of the lack of sufficient rigorous evidence of efficacy.

A few of the better-known safety innovations did find their way onto the report's list of "clear opportunities for research," such as CPOE, "localizing specific surgeries and procedures to high volume centers," and even "improved handwashing compliance." Several of these, such as CPOE and "clinical pharmacist consultation services," also made the list of practices with "medium strength of evidence regarding their impact and effectiveness." Although the 11 top practices have dominated the headlines, 47 of the 73 interventions met the criterion of "medium strength of evidence" or higher.

Keeping in mind that the full report exceeds 600 pages and that its findings are far too complex to summarize easily, the safety community must nonetheless face the fact that only a few of the typical safety practices received the ringing endorsement associated with "greatest strength of evidence." What conclusions can be drawn from this, and what should this report signal to the growing and welcome safety movement in medical care?

The Harvard Medical Practice Study,^{8 - 9} published in 1991, was a cornerstone of the IOM report, has been pivotal in the genesis of the patient safety movement, and serves as an interesting contrast to the focus of the evidence report.

The Harvard Medical Practice Study documented a high rate of iatrogenic injury (3.7%) among hospitalized patients and examined the causes of the adverse events (AEs), and found that about one third of adverse events were currently unpreventable.^{9 - 10} Reducing these AEs would require advances in medical science. The remaining two thirds of AEs were caused by errors in treatment and thus, asserted the study's authors, should be amenable to prevention through sound methods of error reduction. The current safety movement arose as a concerted effort to deal with this problem of medical error and resulting injuries.

The evidence report,² on the other hand, is heavily weighted toward the "other third" of AEs, injuries from care that are not caused by errors. Only 30% (22/73) of the practices evaluated in the report are, in fact, intended to prevent errors. The remainder are mostly technical advances that have been shown to prevent complications (such as use of sterile barriers during catheter insertion, continuous aspiration of subglottic secretions, and use of real-time ultrasound guidance during central line insertion). These are important biomedical advances, and bringing them to attention for much wider implementation will improve patient care.

The evidence report is so heavily weighted toward these technical advances in care for the simple reason that these are the advances that have been studied. That is where the evidence is. For decades, health care research and innovation have focused primarily on biomedical interventions—drugs, devices, and procedures. Advances on these fronts are the classic products of medical research that have led to US preeminence in health care technology.

By contrast, as Shojania et al point out in the report, error prevention—especially the systems issues that underlie a great proportion of patient injury—is a young field, which has commanded the attention of only a small number of researchers and, until recently, has received little funding. The $50 million that the US Congress provided AHRQ in fiscal year 2002 for safety research is less than half of 1% of the National Institutes of Health budget that funds research for these technical and other important medical advances.

In this context, it is remarkable that so many of the error-preventing practices made the evidence-based list at all, and it is not surprising that the experimental data to prove the efficacy of most safety practices are still sparse. It took more than 20 years and repeated huge clinical trials to establish the efficacy of a single procedure, coronary artery bypass graft surgery, at a cost far exceeding the $50 million committed this year for research on patient safety.

The result of this focus on formal scientific evaluation is that the report gives short shrift to implementation of safety practices, the systems changes that health care organizations are struggling with. It focuses more on individual practices. However, the main thrust of the safety movement, one of the most important learnings during the past decade, and the clear message of the IOM report, To Err Is Human, is that safety is primarily a systems problem.^{1 ,11 - 18} On the front lines, this has 2 implications: that better systems must be developed to prevent errors and, equally important, that better systems must be developed to ensure that clinicians provide the effective care they intend to provide.

Thus, achieving safer care has 3 agendas, all of which are necessary for success: identifying what works (efficacy), ensuring that the patient receives it (appropriate use), and delivering it flawlessly (no errors). The relationship among these factors is well illustrated by the safety practice given the number 1 rating in the report: prophylactic anticoagulation for venous thromboembolism. The science, as the report verifies, indicates that in controlled trials this practice reduces the incidence of venous thromboembolism, a serious AE. Anticoagulation is efficacious. However, the practical patient safety issues for practitioners are (1) how to ensure that every patient who needs anticoagulation receives it, and (2) how to ensure that the medication is delivered flawlessly—on time, in the right dose, every time, without fail. Such systems issues are at the heart of safety, but are not addressed by the EPC report by Shojania et al.

The efficacy of systems changes is not as easily tested at the individual patient level as are specific practices, such as subglottic suction or the use of catheters designed to prevent infections, because the rate of failures is often low. Despite the large number of AEs overall, specific types of AEs occur infrequently. Thus, trials based on AEs as the outcome are difficult to conduct and, because the outcomes are infrequent, can be extraordinarily expensive. For example, a high-profile systems change (ie, CPOE) was given a "medium" evidence rating because the trials involved a single institution and because the outcome for which a difference was found was serious medication errors and not adverse drug events (ADEs). Yet, the main trial cost more than $1 million (L.L.L., D.W.B., unpublished data, 1998). A clinical trial in which patients were randomized to CPOE would be difficult and expensive to conduct. This is true of many other systems-level interventions as well. Thus, the traditional evidence-based approach cannot be the sole source of information for advancing patient safety.

Unfortunately, even within its assigned scope of identifying effective practices, the EPC report has some shortcomings that limit its usefulness. First, prioritization is skewed by the strong data bias. As they freely admit, Shojania et al went where the data led them. This is the classic EBM approach, which has worked well in clinical medicine. In addition, most of the reviewers were internists; only a few were safety experts. Understandably, they applied the classic academic medical model rather than the human factors, engineering, or safety theory models that are used in other industries.

The search for formal evaluation data led the authors to concentrate not on the topics of greatest importance to improving safety, but on those for which researchers have recently produced data. As a result, 8 topics account for 44% of the recommendations. For example, one of these, central venous catheter problems, accounts for 7 practices, or 10% of the total. While the safety of central venous catheter use is important, it would not be near the top of the list of challenges for hospital safety officers or near the top of the list of causes of patient harm. Similarly, preventing ventilator-associated pneumonia (5 practices), while important, has limited application.

The effect of skewing is made more apparent by viewing these recommendations through another lens: if the distribution of AEs among hospitalized patients is roughly that suggested by the Harvard Medical Practice Study,^{8 - 9} the interventions in the evidence report would prevent relatively few of all AEs. The evidence report made these prioritizations under great time pressure; however, the impact of the interventions was compressed in such a way that the benefit was not given sufficient attention. The point is not that these practices are unimportant, but that prominence on the list should not be taken as evidence that such practices should necessarily command a high priority for implementation. If the evidence report is taken too literally, that is exactly what will happen.

A second limitation of the report is that key safety practices are omitted. The report downgraded or omitted many established practices because of inadequate evidence, either because of lack of data from controlled trials or because the available data did not prove a reduction in AEs. Accordingly, customary practices, such as sponge counts in surgery, wrong-site surgery prevention techniques, and most anesthesia practices, including intraoperative monitoring, and even the use of checklists, did not make the list.² It would be tragic if these omissions were interpreted as reasons to discontinue these practices.

A good example of accepted safety practices is the group related to prevention of ADEs, which account for an enormous number of hospital AEs.⁹ This field has commanded the lion's share of the safety effort in the past decade and was the most frequent type of AE in the Harvard Medical Practice Study,^{8 - 9} but is represented in only 6 of the practices considered. Important practices, such as implementation of pharmacy-based intravenous admixture systems, removal of concentrated potassium chloride from nursing units, use of computer-generated or electronic medication administration records, educating patients about the safe and accurate use of their medications, and having a hospital pharmacist available on call 24 hours a day, were not included. These practices were omitted from the report because sparse data are available regarding their impact, and in particular they have not been subjected to randomized controlled trials. Yet, there is little question that they are effective methods for reducing ADEs. Some are "trialable," and some are less so. The fact that these practices are not on this list should not lead to any reduction of the important, and successful, efforts being made nationwide to improve medication safety.

Less specific but even more important topics for improving patient safety that Shojania et al considered but did not rate or include in their evaluation tables because they "have not yet received adequate evaluation of their potential health care applications" include the following: "promoting a culture of safety," "use of human factor principles in evaluation of medical devices," "incident reporting," and "refining performance of medical device alarms."

Omission of these topics demonstrates the limitations of requiring the "adequate evaluation" the authors seek for further action on the implied system changes. Does health care's investment in "promoting a culture of safety" really have to await a randomized trial of culture change? Although continued outcomes research is essential for establishing and improving the scientific evidence base for medical practice, formal evidence of the type the evidence report sought is neither appropriate nor essential for all of the interventions needed to improve patient safety. With respect to modern knowledge about how complex systems get safer, the relentless, uncritical requirement of formal scientific proof according to the evidence report standards simply lacks face validity.

Third, a practice was downgraded if it is already widely in use. This seems particularly bizarre. Although the purpose of the evidence report was to highlight opportunities for increased impact, the effect of this arbitrary approach is to give a distorted view of the power of various interventions. Unit dosing is a good example. This practice has the potential to reduce the number of medication administration errors because of the millions of doses given annually. Shojania et al gave it a low rating in part because unit dosing is already widely used in hospitals. However, unit dosing is only partially implemented in most hospitals, and bulk dosing still causes many ADEs, particularly in intensive care units and emergency departments. Achieving nearly 100% unit dosing has been an important challenge for patient safety efforts. The unrealized potential of this practice alone for reducing injury probably far exceeds that of most of the other 72 items on the list.

Fourth, safety, or prevention of AEs, is confounded with other issues involving the provision of optimal care. This is perhaps the greatest anomaly in the evidence report. Six of the 73 practices on the list address 3 problems that have not been considered safety issues: informed consent, end-of-life care, and pneumococcal vaccination in elderly patients. While each is important, and all are practices very much in need of improvement, there is little evidence that any of these 3 is a significant cause of injury. These are important quality-of-care issues, but calling them safety issues muddies both waters, and there are many other equally or more deserving quality issues. Although all safety issues are quality issues, the reverse is not so. The IOM's definition of safety is "freedom from accidental injury." Few would consider community-acquired pneumonia (or measles, for that matter) to be "accidental" injuries.

The policy question posed in the evidence report, and the overriding issue that must now be addressed by the National Quality Forum Committee, is this: "What criteria should be used to determine ‘best practices' for improving patient safety?" We believe that rigorous proof of efficacy according to the rules of the evidence report is neither necessary nor, in many cases, sufficient for recommending widespread use of a safety practice. In addition, as the report commendably points out, it is not always possible to obtain such evidence.

Aviation safety, for example, was not built on evidence that certain practices reduced the frequency of crashes. Instead, it relied on the widespread implementation of hundreds of small changes in procedures, equipment, training, and organization that aggregated to establish an incredibly strong safety culture and amazingly effective practices. These changes made sense; were usually based on sound principles, technical theory, or experience; and addressed real-life problems, but few were subjected to controlled experiments.

In health care, the progress in anesthesia safety is a comparable example. Everyone, including Shojania et al, agrees that the current practice of anesthesia provides an outstanding example of how a high level of safety can be achieved in health care. Anesthesia is the only system in health care that begins to approach the vaunted "six sigma" level of perfection that other industries strive for. Mortality from elective anesthesia has declined 10-fold in the past several decades as the result of a concerted effort to improve safety.¹⁹ This outstanding achievement is attributable not to any single practice or development of new anesthetic agents or even any type of improvement (such as technological advances) but to application of a broad array of changes in process, equipment, organization, supervision, training, and teamwork. However, no single one of these changes has ever been proven to have a clear-cut impact on mortality. Rather, anesthesia safety was achieved by applying a whole host of changes that made sense, were based on an understanding of human factors principles, and had been demonstrated to be effective in other settings. Safety, they showed, is doing a lot of little things that, in the aggregate, make a big difference. In addition, the anesthesia community has measured its progress over time, accumulating a "time series" track record whose signal is virtually incontrovertible. To say that convincing evidence of progress and effect is lacking because randomized trials of all safe anesthesia practices have not been conducted would be Luddite.

These successful experiences in aviation and anesthesia should inform the selection process for best practices in health care. They provide proof of the value of other types of evidence that performance improvement experts find compelling:

An immense body of literature testifies to the value of practices based on principles designed to compensate for human cognitive failings. These include, for example, standardization, simplification, and use of protocols and checklists. Reducing the types of intravenous infusion pumps used in a hospital from 7 to 1 or 2, for example, reduces errors, but it has not been proven in a controlled trial. Is one needed? Hardly. Standardizing medication orders to require the use of leading zeroes, (0.1 mg is acceptable as a dose, but .1 is not), for example, eliminates errors, but has not been proven to reduce ADE rates.

In the leading zeroes example, perhaps only one in a hundred prescriptions in which the zero has been omitted will be misread by a clerk, nurse, or pharmacist as a whole number, and probably less than 10% of those errors will result in a patient injury (1 per 1000 orders), and less than 10% of those will be fatal (1 per 10 000 orders). It would be incredibly difficult to mount a controlled study of sufficient power, of more than 10 000 orders, half with zeroes and half without, to prove that the ADE rate is decreased, or of 100 000 orders to prove mortality has been decreased. Is it necessary? We think not.

A more provocative example is as follows: abundant information exists from many well-performed studies that error rates are substantial in all judgments based on human observations, such as interpretation of radiographs, electrocardiograms, angiograms, and pathology specimens. Two experts will disagree in these interpretations 10% to 20% of the time.^{20 - 21} An effective method for reducing errors is to require duplicate independent readings.^{22 - 27} Is it necessary to document the number of deaths or cases of unnecessary surgery resulting from every possible use of each of these tests to justify establishing duplicate readings as standard practice? We hope not.

The classic and, in medicine, extremely controversial example of practices that are "known" to reduce errors in other environments is limitation of work hours. Hour restrictions are mandated in almost all occupations in which performance failures can affect the lives of others, except health care. Although Shojania et al assign limiting hours of service a low (ie, 4) rating for available evidence, to their credit they included the statement "efforts to reduce fatigue and sleepiness should be undertaken, and the burden of proof should be in the hands of the advocates of the current system to demonstrate that it is safe."² Few safety experts would disagree.

A common theme in all of these examples is that they make sense. To a lay person and to most physicians, these practices seem like obvious things to do. That is basically how aviation and anesthesia made progress: they did what seemed to be the obvious right things to do.^{28 - 29}

The problem with this approach is that sometimes obvious interventions or practices turn out to be wrong. The history of medicine is littered with practices that were discarded when rigorous studies demonstrated they were ineffective or even harmful, such as, for example, the use of lidocaine after myocardial infarction.³⁰ Thus, we do not advocate implementation of safety practices that are based solely on common sense and have no evidence of efficacy. If they were later found to be ineffective, tremendous waste of resources and some potential harm to patients could result. Fortunately, the value of most safety measures has been established directly or by analogous practices in other industries. Many other practices are based on sound human factors principles that are unlikely to be repealed. The National Quality Forum is currently reviewing the evidence report's practices and will be issuing its own recommendations.

However, some effective practices are costly, and it would not be practical to implement all of them. Thus, it is necessary to have a method for prioritizing them whether or not they have been subjected to randomized trials. Methods for prioritizing safety practices should be a key area for future research. Parameters that should be considered include the frequency and severity of the target problem and the evidence that the specific safety practice will be effective, generalizable, and sustainable.

We urge serious consideration of the top-level practices certified by the evidence report.² Some should be recommended as best practices. However, it is crucial to understand that, given its rules for search and judgment, the list in the evidence report is neither a complete nor necessarily an appropriate inventory of practices for priority action to improve patient safety.

Policymakers must consider the entire experience with safety practices, both in health care and in other industries, when deciding which practices should be recommended for widespread use. Evidence from randomized trials is important information, but it is neither sufficient nor necessary for acceptance of a practice. For policymakers to wait for incontrovertible proof of effectiveness before recommending a practice would be a prescription for inaction and an abdication of responsibility. There will never be complete evidence for everything that must be done in medicine. The prudent alternative is to make reasonable judgments based on the best available evidence combined with successful experiences in health care. While some errors in these judgments are inevitable, we believe they will be far outweighed by the improvement in patient safety that will result.