Meta-analysis and Epidemiologic Studies in Drug Development and Postmarketing Surveillance

The unmatched ability of prospectively designed randomized trials to provide unbiased, definitive evidence about the benefits and risks of treatments in the population studied is not in doubt, certain critical aspects of drug therapy, however, cannot be addressed by this method or addressed as rapidly as physicians, patients, and the public would like. In this issue of THE JOURNAL, 2 related articles^{1 - 2} consider the potential value of other methods of discovery of adverse consequences of drug use as well as the beneficial effects of drugs, specifically through epidemiologic methods and meta-analyses (systematic overviews) of data. These methods unquestionably have a place in the assessment of drug treatments but, as has been often pointed out, they must be used with care and recognition of their limitations.

Berlin and Colditz¹ consider the potential uses of meta-analyses, primarily meta-analyses of controlled trials, in the approval and postmarketing evaluation of Food and Drug Administration (FDA)–regulated products. The authors distinguish between prospective and post hoc meta-analyses and between meta-analyses of controlled trials and observational data. With respect to post hoc analyses, Berlin and Colditz correctly note that FDA regulations already require overviews of data on safety and effectiveness in marketing applications.^{3 - 5} More recently, regulations have specifically required that overall data be examined for differential responses in demographic (race, sex, age) subsets, an approach previously urged in FDA guidance documents on studying the elderly⁶ and evaluating drugs in both sexes.⁷ The roles of these overviews are multiple but, with 1 exception (safety evaluation), they are primarily "exploratory" attempts to refine what already is known. Thus, with effectiveness having been established in controlled trials, the overall data are probed to see whether there appears to be consistency from trial to trial and across subsets of the population. In conducting these analyses, investigators must be appropriately aware that subset analyses, whether of single studies or meta-analyses, can be misleading, depending on how many hypotheses are tested. Yusuf et al⁸ have noted that "[T]he overall trial result is usually a better guide to the direction of effect in subgroups than the apparent effect within a subgroup. Failure to specify prior hypotheses, to account for multiple comparisons, or to correct P values increases the chance of finding spurious subgroup effects," and they provide several striking examples of such subset findings that could not be confirmed.

Apparent subset findings usually will need further study. For example, an overall subset effect ordinarily would be examined study by study to look for consistency and it might be necessary to conduct new trials. Nonetheless, the absence of such findings is at least somewhat reassuring regarding the absence of any large difference in response, and results of such analyses often are mentioned cautiously in drug product labeling. Subset findings can be convincing if they are replicable. For example, lesser responsiveness of black hypertensive populations to angiotensin-converting enzyme inhibitors has been observed repeatedly (although not uniformly) and this information appears in labeling for these drugs. Considering the entire effectiveness database in estimating effect size also can mitigate the obvious upward bias of relying only on the trials that achieve statistical significance.

In safety evaluations, meta-analyses are treated as the primary data.^{4 ,9} Because rates of more common adverse events are highly environment dependent, they are usually best estimated by an analysis of pooled controlled trials. It would be most useful, of course, to know why rates vary from one setting to another but, after obvious reasons are explored (eg, dose, underlying disease, sex mix, method of data collection), it is still unusual to be able to explain variability with confidence. With respect to rarer events, the full database is needed to have any reasonable chance of observing events often enough to perform a useful comparison with the control rate.

These "refining" uses of meta-analysis in drug development, looking among the well-controlled studies for further insights and the use of a pooled safety database, are not controversial and are well established. Berlin and Colditz suggest, however, that these post hoc "reactive" approaches are not the best that can be done, and that "meta-experiments" should be a planned component of drug development. They argue first that precise replication of a finding is not as useful as replicating the essential finding in a variety of settings (eg, various age ranges, geographic regions, severities of illness) because the latter approach simultaneously allows greater generalizability and provides the potential ability to look for critical variations. Therefore, they suggest that optimal drug development would consist of planning a series of studies, each addressing pertinent different questions, that would be combined on completion, a "preplanned meta-analysis."

This suggested approach is clearly correct in proposing a variety of studies in different settings. It may not be so different from current drug development methods, however, and it may not always be advantageous to pool results. Any multicenter study, by far the most common kind of efficacy study today, has inevitable site-to-site differences that give it some elements of a meta-experiment. A planned meta-experiment, in which somewhat greater differences are built into distinct trials, does not seem conceptually very different, at least with respect to the basic effectiveness question, and should be interpretable and useful. The studies of effectiveness to support drug approval need not be identical; there is no FDA requirement for, or desire for, exact replication.¹⁰ Indeed, there is a preference for increased variety of settings and exposure. The meta-experiment could perhaps use these variations more effectively to reach a more general conclusion and to identify important differences in response. On the other hand, if studies in well-defined different settings are to provide useful information about results in those settings, each study needs to be large enough to give an independent result. Therefore, it is not clear how much is to be gained by pooling such independently valid studies for analysis as opposed to looking at them separately, except in a few important cases, such as the following.

The ideal situation for a planned meta-analysis would seem to be one in which the individual studies are large enough to examine some end points but not others. One example is in the assessment of demographic subset effects. It is useful to pool results across studies that include all demographic groups when each individual study does not have enough representation of the subset to allow meaningful within-study analysis. Such a pooled analysis seems preferable to separate studies in the subsets because it allows a distinction between subset effects in a common study environment and differences between studies.^{6 - 7} Another example is when there is interest in both low-frequency and higher-frequency outcomes. For instance, in developing a drug for heart failure, it would be important to study various severities of disease, each with somewhat different end points appropriate to the stage of disease. Each study, however, might not be large enough to assess effects on survival satisfactorily, whereas the pooled results of several studies might give useful survival data. There are risks in this approach, of course. If some populations are truly less responsive than others, pooling of data could mask the favorable results in one setting.

Unplanned meta-analyses, post hoc assemblages of randomized trials, pose greater problems of biased selection. Aside from the recognized risk that publication bias may leave the meta-analysis with a biased sample of trials to combine, it is also possible that awareness of the major study results might stimulate the decision to perform a meta-analysis in one case but not another. This is certainly seen in the regulatory environment, where post-hoc proposals to pool previously separate studies is common and clearly data dependent. Moreover, Berlin and Colditz cite a considerable literature expressing concern that the results of meta-analyses and large trials of similar size have been disparate in some cases.^{11 - 12} Therefore, these analyses need to be approached cautiously.¹³ However, individual large single trials may also report unexplained disparate results. The largest trial examining aspirin use following myocardial infarction¹⁴ showed no beneficial effect, in contrast to clear effects in several other good-sized trials. Similarly, 2 large primary prevention trials with aspirin^{15 - 16} reported very different results, one showing a 50% reduction in new myocardial infarctions,¹⁵ the other showing not even a favorable trend.¹⁶ It is thus not necessarily the case that a robust result in a meta-analysis is more likely to be incorrect than the results of a single trial of comparable size, although this question clearly needs further analysis. Lau et al¹⁷ argue that cumulative meta-analyses, had they been conducted, would in some cases have produced results so extreme that continued randomized placebo-controlled trials would have been unacceptable. It is also undeniable that the vast controlled trial databases that can sometimes be assembled for meta-analyses allow outcome and subset inferences that appear to be approachable in no other way.¹⁸

Berlin and Colditz also briefly consider meta-analyses of nonexperimental (observational) studies, but here they are appropriately cautious, not because of doubt about pooling, but because a group of observational studies may share common biases.

The use of epidemiologic methods is considered in more detail in the article by Brewer and Colditz in this issue of THE JOURNAL.² In reviewing methods of postmarketing surveillance for adverse drug reactions, the authors describe the available sources of information and some of the limitations of each. Although they recognize the critical distinction between adverse events that are rare in the population in the absence of drug exposure and those that are common, they do not emphasize sufficiently the important distinction between adverse events for which the frequency is greatly increased by a drug (many-fold), which are comparatively easy to detect using a variety of methods, and events for which the frequency is increased only modestly (for instance, <2-fold). The relative risk is a far more important determinant of how and whether adverse events can be detected than whether the events themselves are rare or common. Changes in the rates of relatively common events are often of greatest concern—a 30% increase in myocardial infarction rates, after all, would be more damaging than a 10-fold or even 100-fold increase in the rate of a 1 per million event—but methods to detect these changes other than through controlled trials are problematic.

When an event is rare in the population and its rate is greatly increased by use of a drug, spontaneous reporting (to detect the existence of the problem) and, in some cases, case-control studies (to establish the magnitude of the effect) can readily document the adverse drug effect and its rate. What is principally needed is for the medical community to notice the event, recognize the possibility that the event is an adverse drug effect, identify the possible drug cause, and report it. This is easiest when the event is typical of drug reactions and has occurred before (eg, agranulocytosis, liver necrosis, Stevens-Johnson syndrome) and when it occurs soon after the initiation of drug use. However, even when the event occurs only after prolonged exposure, or long after exposure if it is very unusual in the absence of drug exposure, it can be detected if someone notices the unusual event, seeks a possible explanation for it, and detects the drug association. There are many examples of such detection, such as vaginal cancer after intrauterine exposure to diethylstilbestrol, pulmonary hypertension after exposure to appetite suppressants, or aplastic anemia due to a variety of agents. In each case, spontaneous reports triggered interest and epidemiologic studies confirmed a great increase in risk in exposed persons.

As Brewer and Colditz point out, the FDA has stimulated a sizable increase in spontaneous reports, with dramatic results. Within the last year, for example, the spontaneous reporting system detected within several months cases of rhabdomyolysis resulting from concomitant use of simvastatin and an inhibitor of its metabolism, mibefridil, and cases of liver necrosis with bromfenac, tolcapone, and troglitazone. The spontaneous reporting system is quite effective for detecting these kinds of rare, serious adverse drug reactions. Rossi and coworkers¹⁹ reported (even when there were far fewer spontaneous reports to the FDA) that there was no added benefit (additional drug reactions detected) from several FDA-requested cohort studies of 10,000 to 20,000 patients. Spontaneous reporting undoubtedly can be improved further. Better data mining techniques would allow signals to be detected earlier and there may be additional sources of signals. Registries of events of interest (eg, liver or marrow toxic effects, specific unusual tumors) also might be useful in detecting rare drug-related events. Better and larger systems linking drug use to hospitalization and outpatient diagnoses might be able to detect rare events and would allow confirmatory case-control or cohort studies to be conducted promptly. When the increased risk caused by the drug is large, these confirmatory methods are very effective.

Events already common in the untreated population (eg, several percent) and for which the rate is increased substantially by drug exposure should be detected in the controlled trial database. What is not yet clear is how to discover modest increases in relatively uncommon events (eg, 0.1%-1%) that are important, such as myocardial infarction, stroke, or death. These effects are not reliably detected in clinical trial databases of usual size. Epidemiologic methods often are used to seek such adverse events, but these methods pose problems, not primarily a problem of study size or "power," as Brewer and Colditz suggest (databases of almost any conceivable size can be found), but the problem of the limited ability of available methods to give reliable answers about modest risks. A 2- to 3-fold relative risk of a myocardial infarction or death is not a "small" increase in risk in the usual sense; it is far larger, for example, than the benefit of such effective treatments as postinfarction aspirin, β-blockade, angiotensin-converting enzyme inhibition, or thrombolysis. Nonetheless, Taubes²⁰ found that a sizable group of epidemiologists did not consider findings of relative risks of this magnitude in epidemiologic studies reliable. Some suggested that replication of such a finding in different environments with different methods might be more persuasive than a single study. This approach would certainly be helpful, but at least 1 frequently cited example of a false association, the 2- to 3-fold increased risk of breast cancer associated with reserpine use,²¹ was observed 3 times in different settings.

Epidemiologic studies, like other nonrandomized trials,²² have several recognized limitations.²³ They often do not use bias-reduction approaches common to randomized trials, such as identifying in advance a small number of specified hypotheses (preferably 1) and a single prospectively specified method of analysis. It is possible that standards for conducting these studies could evolve to lessen these problems, and there has been an attempt to accomplish this.²⁴ Even more difficult is the problem of indication bias or selection bias, ie, the possibility that the patients who received the drug that seemed to cause increased risk of an event already were at greater risk of having that event when the drug was given. This problem is considered at length by MacMahon and coworkers²⁵ in the context of calcium channel blockers, which an epidemiologic study²⁶ suggested might increase the risk of myocardial infarction in hypertensive patients by about 1.6-fold. The authors noted that there is some evidence, as well as a reason to expect, that patients who are prescribed calcium channel blockers are at greater than average risk of coronary heart disease, which could have led to the reported finding even if the drugs had no adverse effect. They also pointed out that randomized trials involving substantial numbers of patients given calcium channel blockers or placebo following myocardial infarction do not show an excess risk of infarction with the calcium channel blockers.²⁷ Where there is the possibility of indication bias, and a modest increase in risk, MacMahon and colleagues doubt that adjustment for such bias will be able to account for all factors, some of which will not be recognized. This view is widely accepted.²⁸

Epidemiologic methods have an impressive track record in detecting larger risks, 3- to 4-fold and greater (eg, estrogen causing endometrial cancer, cigarettes causing lung cancer). Despite uncertainties about the reliability of these methods in detecting more modest risks, there seems little doubt that their use will continue. Several alternatives to the way these studies are conducted and used should be considered. First, the possibility of using more rigorous methods, namely, large simple trials,²⁹ to detect such risks should be considered in some cases (eg, where very wide exposure is expected). It is now apparent that trials in the 40,000-patient range are feasible. If a question is important, answering it well may be worth the cost and effort.

Second, even if an initial epidemiologic study is conducted using currently available methods, it could be considered standard procedure not to report relative risks of less than, perhaps for instance, 2 or 3 until the study is replicated in a different environment. The replication also might take place under more stringent conditions: prospective protocol, single hypothesis, well-described methods of analysis. The disadvantage of this approach would be delayed awareness of what could prove to be a correct observation and perhaps delay in performance of the replications.

Third, scientific and lay media need to understand and deal with the limits of these methods. The problem of contradictory reports to the public on various health and dietary interventions has been described,³⁰ noting, in particular, reports that antioxidants did not prevent colorectal cancer even though earlier reports had indicated they did. A recent study³¹ reporting that dietary fiber does not prevent colorectal cancer updates these disappointments. One suggestion was that the media and the public need to understand better what studies do and do not mean. Although surely that is much to be desired, the understanding of the scientific community seems the more likely candidate for improvement. There is no apparent reason why scientific journals could not impose higher standards on the epidemiologic studies they publish; ironically, they impose the greatest demands on the most methodologically secure studies: controlled trials.³² These higher standards could include better descriptions of methods, including some sense of how many hypotheses were considered, a preference for replication before reports of low relative risks are published, and an insistence on a realistic discussion and appraisal of methodologic limitations. Finally, a worthwhile effort would be a retrospective review of published important epidemiologic findings reporting relatively low relative risks—such as 2 or less—to determine how these results have held up over time.