Users' Guides to the Medical Literature: Title and subTitle BreakXVI. How to Use a Treatment Recommendation

You are a primary care practitioner considering the possibility of anticoagulant therapy with warfarin in a new patient, a 76-year-old woman with chronic congestive heart failure and atrial fibrillation. The patient has no hypertension, valvular disease, or other comorbidity. Aspirin is the only antithrombotic agent that the patient has received over the 10 years during which she has been in atrial fibrillation. Her other medications include captopril, furosemide, and metoprolol. The duration of the patient's atrial fibrillation and her dilated left atrium on echocardiogram dissuade you from prescribing antiarrhythmic therapy. Discussing the issue with the patient, you find she places a high value in avoiding a stroke, a somewhat lower value in avoiding a major hemorrhage, and would accept the inconvenience associated with monitoring anticoagulant therapy.

You have little inclination to review the voluminous original literature relating to the benefits of anticoagulant therapy in reducing stroke or its risk of bleeding, but hope to find an evidence-based recommendation to guide your advice to the patient. In your office file relating to this problem you find a report of a primary study,¹ a decision analysis,² and a recent practice guideline³ that you hope will help.

Each day, clinicians make dozens of patient management decisions. Some are relatively inconsequential, some are important. Each one involves weighing benefits and risks, gains and losses, and recommending or instituting a course of action judged to be in the patient's best interest. These decisions involve an implicit consideration of the relevant evidence, an intuitive integration of the evidence, and a weighing of the likely benefits and harms. In making choices, clinicians may benefit from structured summaries of the options and outcomes, systematic reviews of the evidence regarding the relation between options and outcomes, and recommendations regarding the best choices. This Users' Guide explores the process of developing recommendations, suggests how the process may be conducted systematically, and introduces a taxonomy for differentiating recommendations that are more rigorous (and thus more likely to be trustworthy) from those that are less rigorous (and thus at greater risk of being misleading).

While recommendations may be directed at health policymakers, our focus is advice for practicing clinicians. We will begin by considering the implicit steps that are involved in making a recommendation.

Figure 1 presents the steps involved in developing a recommendation and the formal strategies that are available. The first step in clinical decision making is to define the decision. This involves specifying the alternative courses of action and the alternative outcomes. Often, treatments are designed to delay or prevent an adverse outcome such as stroke, death, or myocardial infarction. In our discussion, we will refer to the outcomes that treatment is designed to prevent as target outcomes. Treatments are associated with their own adverse outcomes—adverse or toxic effects. Ideally, the definition of the decision will be comprehensive—all reasonable alternatives will be considered and all possible beneficial and adverse outcomes will be identified. In patients like the woman with nonvalvular atrial fibrillation in the scenario, options include not treating the patient, giving her aspirin, or anticoagulant therapy with warfarin. Outcomes include minor and major embolic stroke, intracranial hemorrhage, gastrointestinal hemorrhage, minor bleeding, and the inconvenience associated with taking and monitoring medication.

Having identified the options and outcomes, decision makers must evaluate the links between the two—what will the alternative management strategies yield in terms of benefit and harm?⁴ They must also consider how this impact is likely to vary in different groups of patients.⁵ Having made estimates of the consequences of alternative strategies, value judgments about the relative desirability or undesirability of possible outcomes becomes necessary to allow treatment recommendations. We will use the term preferences synonymously with values or value judgments in referring to the process of trading off positive and negative consequences of alternative management strategies.

Recently, investigators have applied scientific principles to the collection, selection, and summarization of evidence, and the valuing of outcomes. We will briefly describe these systematic approaches.

Unsystematic identification and collection of evidence risks biased ascertainment—treatment effects may be underestimated or, more commonly, overestimated, and adverse effects may be exaggerated or ignored. Unsystematic summaries of data run similar risks of bias. One result of these unsystematic approaches may be recommendations advocating harmful treatments and failing to encourage effective therapy. For example, experts advocated routine use of lidocaine for patients with acute myocardial infarction when available data suggested the intervention was ineffective and possibly even harmful and failed to recommend thrombolytic agents when data showed patient benefit.⁶

Systematic reviews deal with this problem by explicitly stating inclusion and exclusion criteria for evidence to be considered, conducting a comprehensive search for the evidence, and summarizing the results according to explicit rules that include examining how effects may vary in different patient subgroups.^{7 - 8} When a systematic review pools data across studies to provide a quantitative estimate of overall treatment effect, we call it a meta-analysis. Systematic reviews provide strong evidence when the quality of the primary studies is high and sample sizes are large and less strong evidence when designs are weaker and sample sizes small. Because judgment is involved in many steps in a systematic review (including specifying inclusion and exclusion criteria, applying these criteria to potentially eligible studies, evaluating the methodological quality of the primary studies, and selecting an approach to data analysis), systematic reviews are not immune from bias. Nevertheless, in their rigorous approach to collecting and summarizing data, systematic reviews reduce the likelihood of bias in estimating the causal links between management options and patient outcomes.

Rigorous decision analysis provides a formal structure for integrating the evidence about the beneficial and harmful effects of treatment options with the values or preferences associated with those beneficial and harmful effects. When done well, a decision analysis will use systematic reviews of the best evidence to estimate the probabilities of the outcomes and use appropriate sources of preferences (those of society or of relevant patient groups) to generate treatment recommendations.^{9 - 10} When a decision analysis includes costs among the outcomes, it becomes an economic analysis and summarizes trade-offs between gains (typically valued in quality-adjusted life years [QALYs]) and resource expenditure (valued in dollars).^{11 - 12} A decision analysis will be open to bias if it fails criteria for a systematic overview in accumulating and summarizing evidence or uses preferences that are arbitrary or come from small or unrepresentative populations (such as a small group of health care providers).

Practice guidelines provide an alternative structure for integrating evidence and applying values to reach treatment recommendations. Practice guideline methodology places less emphasis on precise quantitation than does decision analysis. Instead, it relies on the consensus of a group of decision makers, ideally including experts, front-line clinicians, and patients, who carefully consider the evidence and decide on its implications. Rigorous practice guidelines will also use systematic reviews to summarize evidence and sensible strategies to attribute values to alternative outcomes.^{13 - 14} Guidelines developers may focus on local circumstances. For example, clinicians practicing in rural parts of less industrialized countries without resource to monitor its intensity may reject anticoagulant therapy as a management approach for patients with atrial fibrillation. Practice guidelines may fail methodological standards in the same ways as decision analyses.

We will now contrast these systematic approaches to developing recommendations with historical practice.

Traditionally, authors of original or primary research into therapeutic interventions include recommendations about the use of these interventions in clinical practice in the discussion section of their articles. Authors of systematic reviews and meta-analyses also tend to provide their impressions of the management implications of their studies. Typically, however, individual trials or overviews do not consider all possible management options, but focus on a comparison of 2 or 3 alternatives. They may also fail to identify subpopulations in which the impact of treatment may vary considerably. Finally, when the authors of overviews provide recommendations, they are not typically grounded in an explicit presentation of societal or patient preferences.

Failure to consider these issues may lead to variability in recommendations given the same data. For example, a number of meta-analyses of selective decontamination of the gut using antibiotic prophylaxis for pneumonia in critically ill patients with similar results regarding the impact of treatment on target outcomes resulted in recommendations varying from suggesting implementation, to equivocation, to rejecting implementation.^{15 - 18} Varying recommendations reflect the fact that both investigators reporting primary studies or doing meta-analyses often make their recommendations without benefit of an explicit standardized process or set of rules.

When benefits or risks are dramatic and are essentially homogeneous across an entire population, intuition may provide an adequate guide to making treatment recommendations. Such situations are unusual. In most instances, because of their susceptibility to both bias and random error, intuitive recommendations risk misleading the clinician.

These considerations suggest that when clinicians examine treatment recommendations, they should critically evaluate the methodological quality of the recommendations. The greater the extent to which recommendations adhere to the methodological standards we have mentioned, the greater faith clinicians may place in those recommendations (Table 1). Table 2 presents a scheme for classifying the methodological quality of treatment recommendations, emphasizing the 3 key components: consideration of all relevant options and outcomes, a systematic summary of the evidence, and explicit and/or quantitative consideration of societal or patient preferences. In the next section of the article, we will describe the rating system summarized in Table 2.

Quantitative Summary of Evidence and Values. The most rigorous approach to making recommendations (which we will call a systematic synthesis) involves precisely quantifying all benefits and risks; determining the values of either a group of patients or the general population; where uncertainty exists, making a systematic and quantitative exploration of the range of possible true values; and using quantitative methods to synthesize the data. One approach to meeting these criteria involves conducting a formal decision analysis. Many decision analyses fail to carry out each step in the process in an optimally rigorous fashion; to do so usually requires a major research project.^{9 - 10}

Challenges for investigators doing decision analysis include conducting the systematic reviews required to generate the best estimates of benefits and risks associated with treatment options and measuring how the general public or patients value the relevant outcomes. Typically, a decision analysis values each treatment arm in terms of QALYs. When costs are considered, the decision analysis becomes an economic analysis, and we think in terms of additional dollars spent to gain an additional QALY. The optimal therapy or the cost-effectiveness of alternatives may differ depending on untreated patients' risk of the target outcome.

What a decision analysis or economic analysis usually does not do is to value the benefits, risks, and costs and provide an explicit threshold for decision making. For example, a new treatment might cost $50,000 per QALY gained. Is this a bargain or too great a cost to warrant treatment? Often, investigators doing decision analysis will refer to the cost-effectiveness or cost-utility ratios of currently used treatments to help with this decision. For instance, the decision analysis from the scenario in this article concluded that while the cost of warfarin for patients with at least 1 factor increasing their risk of embolism was $8000 per QALY saved, the cost was $375,000 per QALY saved for a 65-year-old person with no risk factors.² The authors compared these figures to the $50,000 to $100,000 cost per QALY gained when screening adults for hypertension.

Quantitative Summary of Evidence and Values: Explicit Decision Thresholds. Investigators can use the principles of decision analysis to arrive at explicit decision thresholds and present these thresholds in ways that facilitate clinicians' understanding. One such approach involves the number of patients to whom one must administer an intervention to prevent a single target event, the number needed to treat (NNT).¹⁹ Typically, the NNT falls as patients' risk of an adverse outcome rises and may become extremely large when patients are at very low risk. In a previous Users' Guide, we have described the threshold NNT,²⁰ the dividing line between when treatment is warranted (the NNT is low enough that the benefits outweigh the costs and risks) and when it is not (the NNT is too great to warrant treatment). Deriving the threshold NNT involves specifying the relative value associated with preventing the target outcome vs incurring the adverse effects and inconvenience associated with treatment.²¹

Investigators using this approach may also consider costs. If so, they face the additional requirement of specifying the number of dollars one would be willing to pay to prevent a single target event. With or without considering costs, investigators can plug the values they adduce into an equation that generates the threshold NNT.²⁰ They can then look at the risk of the target outcome in untreated subpopulations to whom clinicians might consider administering the intervention. Combining this information with the relative risk reduction associated with the treatment, they can determine on which side of the threshold the treatment falls.

Returning to our example, warfarin decreases the risk of stroke in patients with nonvalvular atrial fibrillation. Since anticoagulation increases bleeding risk, it is not self-evident that we should be recommending the treatment for our patients and must find a way of trading off decreased stroke and increased bleeding. We can calculate the threshold NNT by specifying the major adverse outcome of treatment, bleeding, and the frequency with which it occurs due to treatment. We then specify the impact of these deleterious effects relative to the target event the treatment prevents, a stroke. A variety of studies of relevant patient populations^{22 - 25} suggest that, on average, patients consider 1 severe stroke equivalent to 5 episodes of serious gastrointestinal bleeding. We use these figures to calculate our threshold NNT, which proves to be approximately 152 (Table 3). This implies that if we need to provide anticoagulant therapy to fewer than 152 patients to prevent a stroke, we will do so; if we must provide anticoagulant therapy to more than 152 patients, then our recommendation will be to not treat.

The threshold NNT then facilitates recommendations for specific patient groups. Table 4 summarizes the calculation of the NNT and the associated comparison with the threshold for 2 groups of patients. A meta-analysis of randomized trials tells us that anticoagulant therapy reduces the risk of stroke by 68% (95% confidence interval, 50%-79%) and that this risk reduction is consistent across clinical trials.²⁶ The meta-analysis also provides risk estimates for different groups of patients with strokes. Patients older than 75 years with any previous cerebrovascular events, diabetes, hypertension, or heart disease have a stroke risk of approximately 8.1% per year. Anticoagulation reduces this risk to 2.6% with an NNT of 1 divided by 0.055, or approximately 18 per year. The NNT for this group is appreciably lower than the threshold NNT, suggesting that such patients should be treated.

Patients younger than 65 years with no risk factors have a 1-year stroke risk of 1%, which anticoagulant therapy reduces to 0.32%. The associated NNT of 146 approximates the threshold NNT of 152 and suggests the decision about whether or not to treat is a toss-up.

Clinicians or health care decision makers interested in considering costs in their decisions can look for help from the model. Costs can be included by specifying the dollar value associated with preventing adverse outcomes (for example, Laupacis and colleagues²⁷ have suggested the most that society might be willing to pay to gain a QALY is $100,000). When we consider costs as calculated in the decision analysis from the patient scenario,² we arrive at a threshold NNT of 53, suggesting a more conservative approach to anticoagulant administration (Table 3).

Investigators can use units other than NNT to develop clinically useful decision thresholds. For example, for 81 patients previously treated with cisplatin-based chemotherapy, the average minimum gain in survival that was felt to make the chemotherapy worthwhile was 4.5 months for mild toxicity and 9 months for severe toxicity.²⁸ Such a threshold could be integrated with information about the actual gain in life associated with the treatment to help form the basis for a recommendation about use of cisplatin therapy.

Like other quantitative approaches, considering NNT and the threshold NNT, or alternative thresholds, is intended to supplement clinical judgment, not replace it. Investigators exploring different treatment choices have found the method useful.²⁹ However clinicians use it, the approach highlights the necessity for both valuing the benefits and risks of treatment and understanding the magnitude of those benefits and risks in making a treatment decision.

Quantitative Summary of Evidence, Qualitative Summary of Preferences. Practice guidelines, if they are to minimize bias, should not substitute expert opinion for a systematic review of the literature, and should have an explicit and sensible process for valuing outcomes, an explicit consideration of the impact of uncertainty associated with the evidence and values used in the guidelines, and an explicit statement of the strength of evidence supporting the guideline. When a practice guideline meets these methodological standards, and thereby minimizes bias, we refer to the guideline as evidence-based (Table 1).

Once they have the evidence, investigators and clinicians are often uncomfortable with explicitly specifying preferences in moving from evidence to action. Their reluctance is understandable. Specifying a trade-off between a stroke and a gastrointestinal hemorrhage is not an exercise with which we are familiar. People may feel that identifying a specific value—a stroke is equivalent to 2.5 gastrointestinal hemorrhages, for instance—implies more precision than is realistic. Discomfort may increase further when we specify a dollar value associated with preventing an adverse event.

This may be 1 reason that participants in the development of rigorous practice guidelines, including experts in the content area, methodologists, community practitioners, and patients and their representatives, seldom use numbers to identify the value judgments they are making. Still, a rigorous guideline will establish, reflect, and make explicit the community and patient values on which the recommendation is based.

Most practice guidelines fail to systematically summarize the evidence. Even those that meet criteria for evidence accumulation and summarization do not usually make their underlying values explicit. Guidelines that do not meet either set of criteria produce recommendations of low methodologic rigor.

Practice guidelines that meet the criteria in Table 1 provide an alternative to quantitative strategies to arrive at a systematic synthesis.

Traditionally, investigators provide their results and then make an intuitive recommendation about the action that they believe should follow from their evidence. They may do so without considering all treatment options or all outcomes (Table 2). Even when they consider all relevant treatments and outcomes, they may fail to use community or patient values, or even to make the values they are using explicit. For instance, the authors of a meta-analysis of antithrombotic therapy in atrial fibrillation stated "about one patient in seven in the combined study cohort were at such low risk of stroke (1% per year) that chronic anticoagulation is not warranted."²⁶ Here, the relative value of stroke and gastrointestinal bleeding is implicit in the recommendation. The nature of the value judgment is not transparent, and we have no guarantee that the implicit values reflect those of our patient or community. Clinicians faced with such recommendations need to take care that they are aware of all relevant outcomes, both reductions in targets and treatment-related adverse events, and are aware of the relative values implied in the treatment recommendations.

The unsystematic approach represents the traditional strategy of accumulating and summarizing evidence in an unsystematic fashion and then applying implicit preferences to arrive at a treatment recommendation. The approach is open to bias and is likely to lead to consistent, valid recommendations only when the gradient between beneficial and adverse consequences of alternative actions is very large.

Both quantitative strategies and practice guidelines, when done rigorously, are very resource-intensive. Investigators may adopt less onerous methods and still provide useful insights. Researchers doing meta-analyses may wish to take the first steps in making treatment recommendations without a formal decision analysis or practice guideline development exercise. If they are to optimize the rigor of these tentative recommendations they will comprehensively identify all options and outcomes and use their meta-analysis to establish the causal links between the two. They may then choose to label values in only a qualitative way, such as: "We value preventing a stroke considerably more highly than incurring a gastrointestinal hemorrhage. Given this value, we would be willing to treat a moderate-to-large number of patients to prevent a single target event and would therefore recommend treating all but those at lowest risk of stroke."

Clinicians may find such recommendations useful, and they have the advantage of highlighting that if one does not share the specified values, one would choose an alternative treatment strategy. They may not, however, reflect community or patient preferences. In addition, they are less specific than the process of placing a number on our values. While quantifying values may make us uncomfortable, we are regularly (if unconsciously) making such judgments in the process of instituting or withholding treatment for our patients.

The approaches we have described highlight that patient management decisions are always a function of both evidence and preferences. Clinicians may point out that values are likely to differ substantially between settings. Monitoring of anticoagulant therapy might take on a much stronger negative value in a rural setting where travel distances are large or in a more severely resource-constrained environment in which there is a direct inverse relationship between (for example) the resources available for purchase of antibiotics and those allocated to monitoring levels of anticoagulation.

Patient-to-patient differences in values are equally important. The magnitude of the negative value of anticoagulant monitoring or the relative negative value associated with a stroke vs a gastrointestinal hemorrhage will vary widely between individual patients, even in the same setting. If decisions are so dependent on preferences, what is the point of recommendations?

This line of argument suggests that investigators should systematically search, accumulate, and summarize information for presentation to clinicians. In addition, investigators may highlight the implications of different sets of values for clinical action. The dependence of the decision on the underlying values and the variability of values would suggest that such a presentation would be more useful than a recommendation.

We find this argument compelling. Its implementation is, however, dependent on standard methods of summarizing and presenting information that clinicians are comfortable interpreting and using. Furthermore, it implies clinicians having the time and the methods to ascertain patient values that they can then integrate with the information from systematic reviews of the impact of management decisions on patient outcomes. These requirements are unlikely to be fully met in the immediate future. Moreover, treatment recommendations are likely to remain useful for providing insight, marking progress, highlighting areas where we need more information, and stimulating productive controversy. In any case, clinical decisions are likely to improve if clinicians are aware of the underlying determinants of their actions and are able to be more critical about the recommendations offered to them. Our taxonomy may help to achieve both goals.

The closest statement to a recommendation relevant to your patient from the original journal article¹ is the following: "Many elderly patients with atrial fibrillation are unable to sustain chronic anticoagulation. Furthermore, the risk of bleeding (particularly intracranial hemorrhage) was increased when elderly patients in our study received anticoagulant therapy." This study neither summarized the available evidence nor explicitly stated its underlying values; therefore, we would classify its recommendation as low in rigor.

The decision analysis uses systematic summaries of the available evidence and specifies the patient values used in developing its conclusion that "Treatment with warfarin is cost-effective in patients with nonvalvular atrial fibrillation and one or more additional risk factors for stroke,"² placing it in the high rigor category. Moreover, the patient values used in the analysis appear consistent with your patient's preferences. The only limitation to the decision analysis is that its bottom-line recommendation involves considerations of cost, and you have reservations about including cost considerations in your decision. The practice guideline³ once again uses a systematic summary of the evidence, and, though making frequent reference to patients' values, does not specify the relative value of stroke and bleeding implied in its strong recommendation that high-risk patients such as ours be offered anticoagulant therapy. Nevertheless, armed with consistent recommendations from a systematic synthesis and a recommendation of intermediate rigor, you feel confident recommending your patient begin taking warfarin.