The Hazards of Evidence-Based Medicine: Title and subTitle BreakAssessing Variations in Care

Assessing quality of care frequently involves using measures of processes of care, such as Medicare's 25 quality metrics.¹ Adherence to these processes is thought to lead to improved outcomes. For example, the Surgical Care Improvement Project was introduced in 2006, with the goal of reducing surgical complications by 25% by 2010.² Based on observational studies demonstrating associations between process and outcomes, experts concluded that adherence to this series of process measures would result in better care. Medicare adopted these and published them on its Hospital Compare Web site¹ as measures of hospital quality. However, for some process measures, studies have shown that adherence to these measures is not necessarily associated with improved outcomes. This has been the case for perioperative antibiotic use and postoperative wound infection³ and for acute myocardial infarction, heart failure, and pneumonia.⁴ More worrisome is that in some cases, adherence to the prescribed process measure may be associated with considerable harm, such as with tight glucose control in critically ill patients.⁵

Hence, rewarding institutions with high quality of care as determined by process measures and penalizing systems such as those with poorer performance on these metrics might pose potential harm. For instance, an analysis of geographic variation in health care delivery showed that regional variation in chronic conditions among Medicare beneficiaries was inversely related to their case-fatality rate.⁶ In that analysis, patients in some areas, such as McAllen, Texas, were among those with the highest mean number of chronic conditions, whereas patients in other areas, such as Grand Junction, Colorado, were among those with the fewest. McAllen has been associated with very high Medicare expenditures and poor quality of care based on Hospital Compare, whereas the opposite association has been noted for Grand Junction.⁷ However, although the Medicare expenditures in McAllen are relatively high compared with Grand Junction, the case-fatality rate is lower for a group of patients with more chronic conditions.⁶ The definitive outcome of a lower case-fatality rate in McAllen suggests that the higher expenditures were appropriate, yet the intermediate outcomes from process measures as defined on Hospital Compare¹ suggest the opposite.

Provision of more or fewer health care services relative to the average may be either beneficial or harmful. For individual patients cared for by individual physicians, variation in care is sometimes desirable. One patient's chronic illness is not another’s, and treating all patients the same would be clinical nonsense. However, maximizing variation for individuals does not extend to groups of patients cared for by groups of physicians. Physicians and patients arriving at the same decisions in reference to treatment risks and benefits should average out when outcomes are compared among large groups. Thus, studies of variation in care seek to find similarity between large regions. These studies are limited because the outcomes measured and information available for such comparisons are not always reliable. Before concluding that variation is undesirable, caution is needed when assessing data to ensure that such variation accurately reflects the entire scope of medical decision making and how these decisions affect outcomes.

In this issue of JAMA, Haymart and colleagues⁸ report on variations in the administration of radioactive iodine (RAI) following operations for thyroid cancer. The incidence of thyroid cancer is increasing,⁸ but the reasons for the increase are unclear. Along with the increase in incidence, there has been a concomitant increase in thyroidectomy rates⁹ and an increase in the proportion of patients treated with RAI. Using data from a cohort of 189 219 patients with well-differentiated thyroid cancer treated at 981 hospitals associated with the National Cancer Database, Haymart et al⁸ reported a significant increase in the proportion of patients treated with RAI between 1990 and 2008, although there was wide variation in RAI use, with variation occurring in both directions. On one hand, approximately one-fourth to one-third of patients with advanced-stage disease or large tumors were not treated with RAI despite general agreement that RAI is necessary for these patients.¹⁰ On the other hand, a nearly equal proportion of individuals with low-risk thyroid cancer who probably should not have been treated with RAI did receive this therapy. After adjustment for available patient, tumor, and hospital factors, much of the variance was attributable to unexplained hospital characteristics. The authors conclude that the substantial between-hospital variation in RAI use suggests clinical uncertainty and perhaps inappropriate use of RAI in thyroid cancer management.

In this line of reasoning, conclusions are drawn by assessing variations in care from one hospital or region to another and making assumptions about the appropriateness of that care. However, there is incomplete knowledge about how and why care was delivered in hospitals showing variation. If RAI was not given to high-risk patients, the reasons it was not administered (such as patient preferences) are not captured in the database. If RAI was given to low-risk patients, subtle information regarding a clinician's decision to administer RAI is not captured in these databases. For example, during total thyroidectomy, some surgeons leave a rim of thyroid tissue adjacent to nerves to minimize the risk of nerve injury and rely on RAI to ablate the residual thyroid tissue. Such surgeons have made a conscious risk-benefit decision balancing nerve injury with the risks of RAI. This procedure is coded as a total thyroidectomy in an administrative database and appears in an analysis of that database to be associated with inappropriate administration of RAI. To date, no guideline or clinical trial refutes individual variation for a variety of clinical scenarios in treating thyroid cancer; thus, these scenarios fall into the realm of individual clinician judgment and decision making. Consequently, variation is to be expected. This study highlights the limits of using administrative data, which lack granular information about why decisions are made, to draw conclusions about the appropriateness of care.

Without an assessment of hard, irrefutable measures of clinical decision making that include individual preferences for treatment, decisions about the appropriateness of clinical treatments and variations of care cannot be made. For instance, when comparing McAllen, Texas, with Grand Junction, Colorado, the important outcome of mortality demonstrated that the greater per capita expenditures in McAllen⁷ could be justified because those patients were sicker yet, paradoxically, had lower case-fatality rates than their healthier counterparts in Grand Junction.⁶ An opposite conclusion was drawn when the outcome was quality of care as defined by measures that were not necessarily related to important clinical outcomes; ie, those listed on the Hospital Compare Web site.¹ Similarly, in the report by Haymart et al⁸ on RAI use, without knowing if patients receiving RAI derived benefit or harm, it is difficult to conclude that RAI administration was appropriate or not. In some cases, evaluating intermediate outcomes such as variation in care or adherence to treatment guidelines without linking processes of care to meaningful clinical outcomes may be uninformative or, worse, misleading. The link between processes of care and outcomes must be convincingly demonstrated for proxy measures to be used in assessing quality.

Ultimately, treatment decisions are made by patients, not practitioners and not policy makers. Patients rely on physicians and other health care professionals to deliver the most amount of information possible about treatment risks and benefits. Patients then can consider these issues in the context of their own individual preferences. Policy makers should not interfere with this process unless the evidence they use to mandate care is substantial. Because of uncertainty in the integrity of most administrative databases and registries and the inherent limitation in the amount of information they contain about patient care, policy should only rarely be made based on findings from these sources.