Benefits and Risks of Drug Treatments: Title and subTitle BreakHow to Combine the Best Evidence on Benefits With the Best Data About Adverse Effects

The central theme of the Institute of Medicine report on the US drug safety system was the need for a life cycle approach to drug evaluation: both the benefits and the risks need to be evaluated and integrated during the entire market life of a drug.¹ The Food and Drug Administration Amendments Act of 2007 also called on the agency to improve its methods of communicating risks and benefits to patients and physicians. The Institute of Medicine recommendation to “develop and continually improve a systematic approach to risk-benefit analysis for use throughout the [Food and Drug Administration] in the preapproval and postapproval settings” specifically acknowledges the need for and the challenges of the development of new methods of combining evidence about risks and benefits.¹

Information that combines the best evidence on benefits with the best data on risks is also needed for daily clinical practice. Whenever a patient and physician decide on a particular course of treatment, they do so because they expect that the likely benefits will exceed potential harms. For the benefits of drug treatments, they often have authoritative sources to provide information: randomized trials and systematic reviews and meta-analyses of such trials. For adverse effects, the situation is different. Given the average duration of randomized trials (often months to 1 or 2 years) and the average number of patients in randomized trials (often dozens to a few hundred), such trials are at most able to detect and quantify frequent adverse events that occur only early during treatment. Moreover, the adverse effect has to be known beforehand or anticipated to be recorded systematically in the trials. The study population in trials, which often includes young persons with a single diagnosis and without concurrent disease, is often not representative of those who will eventually use the drug in the community.

The situation does not much improve in meta-analysis: the typical meta-analysis of randomized trials covers 1000 to 2500 individuals,^{2 - 3} only half of whom will have taken the new drug. The sample size precludes good quantification of adverse effects unless they occur at least with a frequency of about 1 per 200 person-years. Meta-analyses of trials do not solve the problem of late adverse effects or the problem of the narrow population included in the trials. The information on harms from trials is incomplete, and the possibility of using and combining such information across trials in systematic reviews is limited.^{3 - 5} Thus, to understand the full spectrum of adverse effects—those that occur late, that were not known beforehand, and that are rare but nevertheless serious—and to be able to investigate the true incidence of known adverse effects in circumstances of actual prescribing, well-designed observational studies will always be necessary. It follows that systematic reviews of drug treatments must include not only the results of randomized trials on benefits but also evidence from observational research on harms.

Although the idea to turn to observational data for evidence may be a surprise, the idea makes sense in view of what randomization does and what randomized trials are good at doing. Randomized trials are uniquely superior to evaluate the benefits, the intended or hoped-for effects of treatments. The random allocation mechanism is able to overcome the strong tendency of physicians to selectively prescribe treatments based on the perceived prognosis and likely outcomes of patients. Although prescriptions that are tailored to individual prognoses constitute desired medical practice, this effort also makes it difficult to compare the benefits of different treatments because patients who receive different treatments will have different prognoses, called “confounding by indication.” In general, data from routine daily practice can therefore not be used to assess the benefits of treatment, especially for the comparison between users and nonusers of a drug.

In contrast, data from routine medical practice may very well be used to investigate adverse effects of drugs.⁶ Adverse effects of new drugs are often unknown and unanticipated when those drugs enter the market. The adverse event is usually unrelated to the condition under treatment. At prescribing, physicians are unable to pay attention to risk factors of an as-yet-unknown adverse effect. Even when an adverse effect, such as hepatic failure, is already established, it may be completely unpredictable or idiosyncratic, or its risk factors may not be known. When risk factors for the adverse effects are known, the potential confounding caused by risk-adverse prescribing can often be countered by suitable restriction and careful choice of a comparison group.

By way of example, consider a study designed to compare the incidence of venous thromboembolism among different types of oral contraceptives. A history of venous thrombosis is an established risk factor for recurrence; hence, oral contraceptives are rarely prescribed to women with a history of venous thromboembolism. For investigation of the risk with different preparations, the solution is to limit the study to first-ever episodes of venous thromboembolism among women who use oral contraceptives. Still, there might be lingering doubts about whether physicians might prescribe different contraceptives to women with potential risk factors for thrombosis. A second part of the solution is to further limit the study population to those women who have no known risk factors for first venous thrombosis. In women so selected, the event would have been impossible to predict at prescribing.⁷ Restriction can be a powerful weapon in observational research on harms.⁸ Such restrictions often work best when different drugs with similar indications can be compared.⁹ These kinds of carefully designed approaches permit the use of data from daily medical practice to investigate the harms of treatment in a scientifically sound way: by ensuring comparability. Data of sufficient quality about outcomes, exposures, and covariates are essential for these analyses.

A comparison of the findings, for the same treatment and the same adverse effects, between large meta-analyses of randomized trials and large observational studies testifies to the credibility of observational research about harms.¹⁰ In a study of 15 drug-harm topics, there was evidence about the frequency of a particular adverse effect from meta-analyses of randomized trials, comprising in total at least 4000 patients, and there was similar evidence, on the same harm with the same drug for the same indication, from observational studies comprising at least 4000 patients. Neither design led to consistently higher or lower measures of effect sizes. If anything, observational studies yielded lower absolute excess risks of harm than randomized trials. A likely explanation is that observational studies are based on usual practice data, with more noise in the data recording, leading to nondifferential misclassification that biases estimates toward the null. In the few instances in which observational studies yielded much larger relative risks than randomized trials, the observational data were likely to reflect actual prescribing to a less selected group of patients than had been enrolled in the trials; for example, for intracranial bleeding as an adverse effect of oral anticoagulation, it is likely that patients in the trials were strongly selected in a risk-averse way. The observational data are a better reflection of the frequency of harms experienced by patients in actual practice.^{10 - 11} Clinical trial data and observational evidence really complement each other.¹² A dramatic example is the interaction between selective serotonin reuptake inhibitors and nonsteroidal anti-inflammatory drugs that was missed in selective serotonin reuptake inhibitor trials but led to an excess of upper gastrointestinal bleeding in combined users: The number needed to harm from a meta-analysis of observational studies was close to 1 in 100.¹³

The greatest challenge ahead is that 2 disciplines should be combined: the world of systematic reviews that uses tight protocols to retrieve and pool evidence about benefits from randomized trials and the world of pharmacoepidemiology that uses observational epidemiology to identify harms and is often deeply preoccupied with the finer points of causal reasoning. The 2 fields are populated by different individuals who publish in different journals and use different textbooks and approaches. Epidemiology uses techniques such as confounder scores, propensity scores, or instrumental variable analysis that are unnecessary in the analysis of data from randomized controlled trials. Reviews about harms will often take the form of deliberations about likely biases and confounders and how these were met, or not, in various studies. That information is an essential part of the evidence base.

The challenge of combining randomized trial evidence on benefits, with a mixture of randomized and observational evidence on harms, was recently addressed in guidelines from the Agency for Healthcare Research and Quality.¹⁴ These guidelines clearly separate the use of observational evidence for beneficial effects, for which the possibilities are scant, and the use of the same type of evidence for harms, for which the possibilities are rich. In several examples published under the aegis of the agency, the benefit side rests on data from randomized trials and the harms side on a mixture of randomized and observational evidence, often mainly the latter. Also, the latest version of the Cochrane Handbook for Systematic Reviews on Interventions¹⁵ has extensive chapters on harms, as well as on nonrandomized studies, but it does not yet clearly distinguish the different uses of observational data. In a study of Cochrane reviews, Hopewell et al³ found that the reviews made little use of observational research for harms and concluded that this oversight might lead to inadequate reviews.

The integration of randomized and observational evidence to estimate harms of medical treatments may indeed be a novelty for many scientists involved in systematic reviews, as well as for many practitioners of pharmacoepidemiology. However, for a future that combines benefit and harms assessment, systematic reviews will need to incorporate and integrate the best information from both randomized trials and observational studies. Working in collaboration with all parties involved, the medical and scientific agencies in the United States and Europe can take the lead in efforts to improve systematic integration of information about risks and benefits of drug treatments.