Institute of Medicine Medical Error Figures Are Not Exaggerated

Few publications in recent memory have received as much notice or stimulated as swift a response among policy makers as the Institute of Medicine (IOM) report on medical errors.¹ Within 2 weeks of the report's release last November, Congress began hearings and the president ordered a government-wide study of the feasibility of implementing the report's recommendations. The IOM called for a broad national effort to include establishment of a Center for Patient Safety within the Agency for Healthcare Research and Quality, expanded reporting of adverse events and errors, development of safety programs in health care organizations, and intensified efforts by regulators, health care purchasers, and professional societies. However, while the objective of the IOM report, and the thrust of its recommendations, was to stimulate a national effort to improve patient safety, what initially grabbed public attention was the declaration that between 44,000 and 98,000 people die in US hospitals annually as a result of medical errors. These estimates represent current national extrapolations from the results of 2 large population-based studies carried out to assess the impact of medical injury.^{2 - 3}

A telling measure of the impact of this publicity has been the sudden emergence of questions about the validity of the mortality estimates, particularly those from the first Medical Practice Study (MPS). These findings have not been seriously challenged since they were published 9 years ago. In this issue of THE JOURNAL, McDonald et al⁴ raise new questions about these mortality figures and their implications, noting that the MPS investigators did not calculate "excess" mortality, and may, therefore, have exaggerated the extent of fatal injury.

The first MPS did, in fact, have significant limitations and important methodological weaknesses. It was not designed primarily to study risk factors for injury, for example, but to assess the extent of injury that could lead to malpractice litigation, hence the exclusion of nondisabling injuries and the focus on negligence. The study's most serious limitation is probably that it was a retrospective medical record review study. Many important events in patient care are not recorded in the medical record. Some errors are not even known to clinicians caring for the patient. Studies of autopsy, for example, have found potentially fatal misdiagnoses in 20% to 40% of cases.^{5 - 7} On balance, the reliance on information extracted from medical records most likely led to a substantial underestimate of the prevalence of injury.

Another serious weakness of the MPS is that it relied on implicit judgments by physicians. While extensive efforts were made to strengthen the accuracy and reproducibility of these judgments through training of physician reviewers, use of a highly structured data collection instrument, and duplicate review with rereview and resolution of disagreements, errors undoubtedly occurred. It is possible that these errors "canceled out," ie, overinterpretation of medical error was balanced by underinterpretation, but that is unknown. A serious weakness of any retrospective review is hindsight bias, the tendency to impute causation to an action when the (bad) outcome is known.⁸ Hindsight bias would tend to overestimate the number of deaths due to adverse events.

McDonald et al also state that many patients categorized in the MPS as dying as the result of an adverse event would have died anyway, and that the more relevant number is those who died solely because of the adverse event. To estimate this "excess mortality" they determined the mortality among all patients whose records were screened for adverse events in the MPS. They label these patients selected for chart review the "high-severity group" and then calculate mortality for this group overall.

Unfortunately, this group of patients is not a high-severity group; it was chosen in the MPS not because these patients had complicated illnesses or other risk factors for sustaining an adverse event, but because they met 1 or more screening criteria, events that sometimes result from an adverse event. In fact, the screened group included many patients who were not very sick (Table 1). In addition, a large proportion of the patients in the sample who were severely ill or had complicated conditions were not among those who met screening criteria. For example, all patients who had major surgery, acute myocardial infarction, pneumonia, or stroke who had an uncomplicated course (and therefore did not meet screening criteria) were excluded, as were patients who were admitted for planned terminal care, had a do-not-resuscitate order, or were extremely ill. Even many intensive care unit patients did not meet any of the screening criteria.

The problem in defining a control group arises from equating screening criteria with risk factors, in particular, risk factors for dying. Screening criteria are not risk factors. While both risk factors and screening criteria are "predictors," the implications of those predictions are very different. Risk factors are characteristics that increase the likelihood of experiencing a certain undesired outcome. For example, having diabetes increases the likelihood that a person will develop coronary artery disease later. The term also implies that the factor causes, at least in part, the outcome in question, although the mechanism may be totally obscure. Screening criteria, on the other hand, are indicators or markers of an undesirable outcome that has already occurred. They are consequences, not causes, and they refer to past not future events. Thus, "death" can be a screening criterion, but not a risk factor (obviously) for death. The relationships in screening criteria are the reverse of those for risk factors, both in time and possible causation. A myocardial infarction indicates that a patient may have diabetes, but the infarct does not cause diabetes. The screening criteria used in the MPS were not predictors of future events, including death; they were markers, outcomes that could have resulted from an adverse event that had occurred. In fact, 10 of the criteria were adverse events. Thus, they cannot serve to define the control group for the analysis.

The second problem with equating screening criteria and risk factors is that death becomes both a predictor and an outcome, and it appears in both the numerator and the denominator. One cannot predict an outcome with itself. The logical fallacy is evident in the extreme example. Suppose, for example, the MPS investigators had elected to use only 1 screening criterion: death. The mortality in the control group would be 100%, and if adverse events were identified in 12% of cases, the adverse event mortality would be 12% in this group.

Even if the high-severity group (minus deaths) could be used as an appropriate comparison group, it is not appropriate to attribute all deaths in the sample (1069 by their calculation) to this group, since most of these patients did not meet the other screening criteria. What is needed is the death rate for patients who met the screening criteria other than death. Unfortunately, these data were not collected. Patients had to meet only 1 of the screening criteria, including death, and we do not know whether patients who died met other screening criteria.

While it is not possible to calculate the excess mortality attributable to adverse events by "factoring out" the baseline death risk using the MPS data, and the number may have been inflated by reviewers' hindsight bias, examination of the case descriptions does not suggest that they were spurious findings. Nor were many of the deaths inevitable absent an adverse event. Some seem to have the impression that many of the deaths attributed to adverse events were the result of minor incidents in severely ill people, many of whom would have died anyway. This is not so. First, as noted, terminally ill patients were excluded from the study. Review of the cases in the subset of negligent deaths (88% of the total attributed to error) reveals 2 groups of patients: a small group, 14%, who were severely ill and in whom the adverse event tipped the balance; and a larger group, 86%, for whom the error was not a superimposed event, but a major factor leading to the patient's death (L.L., unpublished data, 1999). Examples of the latter include a cerebrovascular accident in a patient with atrial flutter who was not treated with anticoagulants, overwhelming sepsis due to spontaneous rupture of the intestine in a patient with signs of intestinal obstruction that was untreated for more than 24 hours, and brain damage from hypotension due to blood loss from unrecognized rupture of the spleen.

But what are the ethical implications of this search for "excess" mortality? Does the fact that some patients would have died anyway somehow lessen the significance of their deaths? Not for the patient, his/her family, or for anyone who faces hospital admission. But it does for many physicians, and the reason is instructive.

Knowing that some of the patients "would have died anyway" is important for physicians because it lessens the burden of guilt. Physicians feel responsible for deaths due to errors, which is appropriate and key to physicians' professionalism. But we also feel shame and guilt, which is inappropriate and misguided, since errors are rarely due to carelessness. Failure to understand this is to miss the main message of the IOM report, which is not the mortality figures, but the admonition encapsulated in its title, To Err Is Human.

The transforming insight for medicine from human factors research is that errors are rarely due to personal failings, inadequacies, and carelessness. Rather, they result from defects in the design and conditions of medical work that lead careful, competent, caring physicians and nurses to make mistakes that are often no different from the simple mistakes people make every day, but which can have devastating consequences for patients. Errors result from faulty systems not from faulty people, so it is the systems that must be fixed. Errors are excusable; ignoring them is not.

Three reasons suggest that the IOM report did not exaggerate the extent of medical injury and death. First, despite the limits of record reviews, it is unlikely the reviewers found adverse events that did not exist. However, they undoubtedly missed some that did occur because many adverse events and errors are never recorded in the medical record, either because they are concealed or not recognized. Other errors are discovered after the patient is discharged. In fact, in the MPS, an additional 6% of hospital-caused adverse events were discovered after discharge, but were excluded from the analyses because they were an unknown fraction of all such events.² Therefore, any record-review study produces at best a "lower bound."⁹

Second, neither of the large studies examined the extent of injuries that occur outside of the hospital. More than half of surgical procedures (numbering now in the tens of millions) take place outside of a hospital setting,¹⁰ and the adverse event rates for these procedures have not been studied. Even if complication and death rates are much lower than in-hospital care, the absolute numbers must be substantial, as suggested by the recent report of deaths associated with liposuction.¹¹

Third, when prospective detailed studies are performed, error and injury rates are almost invariably much higher than indicated by the large record-review studies.^{12 - 13} In a large study of patients who died from acute myocardial infarction, pneumonia, or cerebrovascular accident (conditions that account for 36% of all hospital deaths¹⁴ ), DuBois and Brook¹⁵ found that 14% to 27% of deaths were preventable. Andrews et al¹⁶ found that 17% of intensive care unit patients had preventable serious or fatal adverse events. The Centers for Disease Control and Prevention estimates that 500,000 surgical-site infections occur each year.¹⁷ One large controlled study found the excess mortality rate of surgical-site infections to be 4.3%, suggesting 20,000 deaths annually from this cause alone.¹⁸ These data are strong evidence that record-review studies seriously underestimate the extent of medical injury.

The IOM report has galvanized a national movement to improve patient safety. It is about time. Although the initial impact of the IOM report is in part due to the shocking figures (which, unfortunately, are not exaggerated), its long-term impact will result from the validity of its message that errors can be prevented by redesigning medical work. Rather than attempting to assuage guilt or outrage about errors by punishing, discounting, or self-flagellation, physicians need to look to preventing recurrence of errors. Errors and "excess" mortality can be eliminated, but only if concern and attention is shifted away from individuals and toward the error-prone systems in which clinicians work. That is the IOM message, and it is a hopeful one. Physicians should embrace this message with enthusiasm and vigor.

Submit Online Comments: We invite readers to submit comments regarding these articles to be posted online. Please submit comments related to these articles along with your name, professional degree(s), affiliation, and any conflicts of interest to jamaonline@ama-assn.org. Decisions regarding posting and editing of comments will be made solely by JAMA editors.

Read Online Comments