Evidence-Based Medicine Requires Appropriate Clinical Context

What if a patient—after diagnostic tests have been performed and there is no more certainty to obtain—still has a 1 in 100 chance of having venous thromboembolism (VTE)? Should the patient's physician engage the patient in a discussion of the harm and benefit of anticoagulation? What if the chance of VTE was 1 in 20? Or even 1 in 10?

Most clinicians may have difficulty answering these questions because of the need for judgment about treatment when certainty is impossible. This judgment has to balance the absolute benefit against the absolute harm for treating a disease when there is uncertainty that the disease exists. This concept reflects a treatment threshold¹ representing the probability of a disease, such as VTE, above which treatment would provide more benefit than harm, and below which treatment would produce more harm than benefit. If this probability is not known, and if variations in the individual patient's probabilities after testing is completed are not considered within the clinical practice context, physicians will have difficulty applying evidence-based medicine to patient care.

Any recommendation about the use of diagnostic testing must begin with a judgment about the value of treatment once the diagnosis is made. Diagnosis and treatment are so intimately linked that both should be part of the analysis of the utility of any diagnostic test. For example, testing for some diagnoses for which treatment is not helpful and, additionally, carries significant harm, is not a good idea. An example might be testing to find cancer in a patient with a poor prognosis from another condition that would preclude treatment for the cancer. Also, testing for a diagnosis that has an effective and beneficial treatment but with little associated harm or cost is also not good clinical care, because in this case, treatment may always be best. An example might be testing for causes of a sore throat in a patient with a high probability of a treatable infection for which no set of negative tests would dissuade a clinician from recommending treatment.²

In this issue of JAMA, Johnson et al³ report the results of a meta-analysis of 7 studies assessing the occurrence of VTE in 4731 patients with a single negative whole-leg compression ultrasound (CUS) result who did not receive anticoagulation; among these patients, there were 34 VTE events or suspected VTE-related deaths. The authors concluded, “withholding anticoagulation following a single negative whole-leg CUS result was associated with a low risk of venous thromboembolism during 3-month follow-up.”³

A summary estimate from a meta-analysis reflects an “average” risk, obscuring the clinical reality of variations in probabilities. It is unlikely that any single patient in any of the studies included in this analysis had exactly the probability of VTE after a CUS of 0.57% (95% confidence interval [CI], 0.25%-0.89%). Accordingly, using this average probability for clinical decision making in some clinical contexts may do more harm than good.

Among the 7 studies included in the meta-analysis by Johnson et al,³ the range of probabilities for developing VTE within 3 months following a negative CUS result varied from 0% to 3.6% (see Figure 2 in the article). Three of the 7 studies had a greater than average probability. In meta-analysis weighting techniques, studies that have small standard errors, usually from large sample sizes, have greater influence on the summary estimated effect size. For example, one study by Sevestre et al,⁴ which contributed 1243 patients to the analysis and reported a VTE incidence of 0.48% (95% CI, 0.18%-1.05%), appears to heavily influence the overall pooled estimate of posttest probability for VTE, but included only ambulatory patients. In contrast, the authors also included data from another study by Sevestre et al, which contributed 513 patients to the meta-analysis; in that report, the probability of VTE after a negative CUS result was 1.95% (95% CI, 0.94%-3.56%), but the study was performed in hospitalized patients. While it is not clear from the overall report, it is likely that the risk for developing VTE in hospitalized patients was higher than that for ambulatory patients.

These wide ranges of VTE probability estimates are due to the variable clinical contexts and heterogeneity in patient characteristics. Reporting summary estimates and 95% CIs is not an ideal way to produce and report evidence, at least not if the goal is to provide guidance to clinicians regarding treatment strategies. A more helpful report would include the variability in the clinical scenarios physicians encounter and explain how these pertain to outcomes that can be inferred from the meta-analysis. Johnson et al³ do provide event rate estimates based on pretest probabilities according to the Wells score⁵ for the subset of patients (n = 1618) for whom individual patient data were obtained. The pooled VTE incidence rates for the low, moderate, and high pretest probability groups were 0.29% (95% CI, 0%-0.70%), 0.82% (95% CI, 0%-1.83%), and 2.49% (0%-7.11%), respectively, with similar results obtained using regression models.

To be most useful, evidence from clinical research studies must relate to a specific clinical context by more explicitly addressing variations in those clinical contexts that are relevant to individual patients. Generalizing the findings related to a diagnostic test or treatment regimen beyond the specific context from which a study was performed is fraught with danger. For instance, based on the meta-analysis by Johnson et al,³ clinicians may infer that not initiating anticoagulation treatment after a negative CUS result in some surgical or ambulatory patients at low risk of having VTE may be appropriate; however, that inference may not may not be true for hospitalized patients or those with cancer. Greater detail about individual patient scenarios is necessary to facilitate better application of the study results to individual patients. One helpful approach may be for reports of meta-analyses to include, in detail, the inclusion and exclusion criteria for patients enrolled in the original studies.

However, summary statements from meta-analyses should not be used to guide patient care. Such conclusions are not helpful when the clinical studies are combined and averaged in a way that reduces the complex world of medical care to overly simple and consequently not clinically useful statistical summaries. If scientific reports continue to emphasize statistical averaging without proper clinical context, it will be difficult to advance the care of individual patients. Moreover, meta-analysis may have a useful role in synthesizing available evidence, especially, for example, in identifying signals of potential harm that may not be readily apparent in individual studies.^{6 - 7} However, meta-analyses are most appropriately used to formulate, but not test hypotheses.⁸

Evidence-based medicine must evolve to include clinically and contextually explicit estimates for outcomes as guides to care. Clinical trials and studies evaluating diagnostic tests should be designed and reported to enable clinicians to maximize the care of individual patients, so they can avoid doing the sometimes right things for the sometimes wrong patients.