Looking for Medical Injuries Where the Light Is Bright

Health care quality improvement experts often argue that "you can't manage what you can't measure." Suitable yardsticks are essential to judge the magnitude of potential quality problems and track whether interventions improve care. However, this aphorism needs one critical addendum: "You can't measure what you can't define."

Measurement and definitional issues loom large when discussing patient safety. The bellwether 1999 Institute of Medicine report To Err Is Human provided compelling evidence that medical errors pose daily risks throughout the US health care system but failed to quash controversy about the magnitude of that risk.¹ The best-known estimates of the extent of medical error rely on extrapolations from medical record review studies,^{2 - 3} although these numbers have generated heated debate.^{4 - 6}

Delineating definitions, though, should precede measurement. Another Institute of Medicine report defined safety as "avoiding injuries to patients from the care that is intended to help them."^{7 (p39)} Producing useful patient safety measures (ie, measures that can assist in managing and improving care) requires honing this definition to the subset of events that are amenable to improvement. While most observers agree that iatrogenic injuries occur in virtually all practice settings, attribution of injuries to error is complicated. Medical harm can result from myriad and sometimes intertwined factors, including the natural history of patients' diseases, coexisting medical conditions and risk factors, access to and availability of care, recognized toxic effects of appropriate therapies, clinical judgments and misjudgments, flaws in executing medical interventions, and bad luck. Although injuries sometimes trace back clearly to actions of individual practitioners, experts believe that multiple deficiencies latent in complex care delivery systems contribute importantly to most preventable iatrogenic injuries.⁸

To be maximally useful, patient safety measurement tools should therefore focus on preventable injuries. Preventable injuries, in contrast with complications resulting from recognized risky therapies administered correctly, offer actionable targets for quality improvement. Unfortunately, preventable injuries are technically difficult and expensive to capture. Chart reviews, although rich in clinical detail, are expensive, fail to identify undocumented events and causes, and often produce unreliable judgments about preventability.⁵ Incident reporting systems, required by many state health departments and accreditation bodies, are rarely used by physicians and undercount incidents by at least 1 order of magnitude.⁹ Sophisticated computer algorithms that monitor clinical databases and electronic medical records offer promise, but existing algorithms are typically insensitive screens, and few hospitals have adequate resources to develop or implement these high-end applications.¹⁰

Given this situation, exploring the utility of existing administrative data, such as computerized hospital discharge abstracts, for measuring patient safety seems reasonable. Administrative data offer significant attractions, including low cost, ready availability, and coverage of large populations. Creative combinations of administrative data elements, including diagnosis and procedure codes (along with procedure dates), could yield insight into clinical events or conditions that might represent safety problems. Recognizing this opportunity, the US Agency for Healthcare Research and Quality (AHRQ) commissioned researchers at the University of California, and Stanford University to update and expand a set of measures to identify potential safety problems. Informed by literature reviews, expert opinion, analyses of administrative databases,¹⁰ and the Complications Screening Program (CSP) that we developed in the early 1990s,^{11 - 12} the investigators created 20 Patient Safety Indicators (PSIs) by linking discharge diagnoses and procedure codes with other information from computerized hospital discharge abstracts.^{13 - 14} The AHRQ has usefully posted this computerized algorithm on its Internet site, freely available to all, calling the PSIs a "quick checkup" that "can help hospitals enhance their patient safety performance by quickly detecting potential medical errors in patients who have undergone medical or surgical care."¹⁵

In this issue of THE JOURNAL, Zhan and Miller¹⁶ analyze excess mortality, length of stay, and charges among inpatients who had 1 of 18 PSI-defined incidents identified using administrative data from a large, representative sample of US hospitalizations. Using sophisticated analytic methods, the authors estimate that these 18 types of medical events may account for 2.4 million extra hospital days, $9.3 billion in excess charges, and almost 32 600 attributable deaths in the United States annually. They argue that the results represent only the "tip of the iceberg," given the limited number of conditions included in the PSI algorithm and problems inherent in the use of administrative data. The authors carefully note the limitations of their work and recommend that the results be used advisedly. Little is known about the validity of PSIs to compare hospitals and communities, to track changes over time, and to measure preventable complications.^{13 (p145)} Nonetheless, given their staggering magnitude, these estimates are clearly sobering and provide a more granular view of specific types of complications than typically presented.

Today's urgent social need to understand patient safety makes it sensible to use existing administrative data sources to assess the problem. Like the man searching for his lost keys under the lamppost, clinicians and quality improvement professionals might reasonably start looking for potential answers where the light is best. But what do the PSIs actually mean? Specifically, do the PSI measures capture actionable patient safety problems (ie, preventable iatrogenic events) amenable to improvement or complications resulting from other causes? How sensitive are the PSIs for identifying substandard care? Findings from a multiyear, AHRQ-funded project involving 1298 in-depth medical record reviews to validate the CSP—a progenitor of the PSIs—give some pause.

The results of the CSP validation study, described in detail elsewhere,^{17 - 21} suggest that administrative data–based algorithms provide questionable insight into substandard hospital care. Briefly, the study failed to find objective clinical evidence in the medical record to support hospital-assigned discharge diagnosis codes used to identify complications for 19% of surgical and 30% of medical admissions.¹⁷ Discharge diagnoses used by the CSP to flag complications represented conditions that were present on admission for 13% of surgical and 58% of medical cases (ie, these conditions had not occurred during the hospitalization from iatrogenic causes).¹⁸ Although physician reviewers confirmed the presence of CSP-flagged complications among 68% of surgical and 27% of medical patients, they found quality problems in only 30% of surgical and 16% of medical cases.¹⁹ Nurse reviewers using structured chart review instruments and physicians conducting reviews based on their implicit clinical judgment frequently disagreed about the presence of quality problems in cases flagged as complications by the CSP.²⁰ Indeed, an expert panel found it extraordinarily difficult to construct review instruments to identify specific process-of-care problems (ie, actions by clinicians) that might contribute to the occurrence of many complications. The conclusion was that the CSP was a useful screening tool for selecting certain surgical (but not medical) inpatient cases for further, detailed chart review but that it offered little information about quality-of-care deficiencies.

Straightforward strategies exist to enhance the ability of administrative data to identify in-hospital complications of care. Attaching a flag to each discharge diagnosis indicating whether it was present on admission is an obvious example, already implemented in New York and California. However, widespread use of methods like the PSIs could have untoward consequences. If PSI results are used to evaluate individual hospitals, shifts in discharge diagnosis coding may occur (ie, hospitals could stop coding certain conditions that arise in the hospital). Monitoring and redressing such changes in coding would be cumbersome and expensive, as Medicare has found in overseeing coding related to assignment of diagnosis related groups for hospital payment.

Ultimately, as the Institute of Medicine noted,¹ measuring preventable harm will require a comprehensive and far-reaching approach. Many inpatient injuries and errors that appear most prevalent based on chart review studies, such as adverse drug events, diagnostic errors, and problems with adequate monitoring and follow-up, require sources other than administrative data. Capturing safety information reliably and efficiently may become easier with electronic monitoring algorithms of clinical databases and computerized text searches of electronic medical records.²² In addition, deploying enhanced incident reporting systems based on the aviation model and developing intensive surveillance of high-risk processes and practice settings may be worthwhile investments.

The PSIs are a step in the right direction, making transparent and accessible information that may someday help to prevent errors and injuries. However, the precise meaning of the PSIs and other administrative data–based measures remains elusive. Developing and validating a robust set of measurement tools is essential to move patient safety information out of the shadows and into the light.