The Role of Meta-analysis in the Regulatory Process for Foods, Drugs, and Devices

The synthesis of research findings, long a central part of the scientific process, offers the potential to identify areas of agreement in a field of science, areas of discrepancy, and areas that require further research. In this article, we focus on issues arising from the application of research synthesis in the formulation of drug, device, and other regulatory and approval policies. For purposes of discussion, we shall assume that the quality of the meta-analysis is not an issue.^{1 - 2} Rather, we explore the possible role of meta-analysis in the regulatory process, focusing on, but not limiting discussion to, the drug approval process. As the regulatory requirements differ for drugs, foods, and devices, some comments may not apply equally across all areas. We want to stimulate debate and discussion and perhaps identify areas that may be resolved either through future research or at the policy level.

Replication (or other substantiation) of findings is a long-standing requirement for drug approval that draws on the need to rule out chance findings and on the underlying belief that replication is a means of validating results.³ In an authoritative review, Lindsay and Ehrenberg³ argue that "repetitions need not and should not be mere repetitions." Rather, "they can be designed to extend the scope of previous results, so as to lead to more powerful empirical generalizations." That is, we gain strength in inference when the range of patient characteristics has been broadened by replicating findings in studies with populations that vary in age range, geographic region, severity of underlying illness, and the like. Before moving to a specific discussion of meta-analysis in the regulatory process, we summarize briefly a few of the other arguments made by Lindsay and Ehrenberg.³

By definition, conditions are never identical across replicate trials, even if only the actual patients included and the calendar time of their performance changes. A truly identical replication should produce identical results. The strength of evidence comes from the fact that the same result is obtained despite the differences in conditions. Close replication means trying to keep nearly everything the same or similar (eg, populations, sampling procedure, measurement techniques). It is particularly important early in a program of research to establish quickly, relatively easily, and cheaply whether a new result can be repeated at all. From the perspective of drug or device development, the sponsor may begin to replicate the study under optimal conditions with the group of patients in whom the drug is most likely to be effective (as determined in phase 1 and phase 2 studies).

As a therapeutic area becomes better understood, differentiated replication, with deliberate or at least known variations in fairly major aspects of the conditions of study, increases in value. Consistent results mean the finding is generalizable; varying results teach us what conditions lead to successful treatment. Heterogeneity of patient characteristics and protocols should be built into the research plan in a carefully controlled manner.

We discuss below the use of planned meta-analysis as an element of the approval process. What we mean by a planned meta-analysis is not just planning the logistics, but planning the scientific questions to be addressed. That is, sponsors should be planning studies in advance, to be combined upon completion, with enough similarity at least in terms of data collected to allow combination of results across studies. Ideally, this type of planning is guided by discussion with the Food and Drug Administration (FDA) as to what studies and analyses are acceptable and appropriate. Methods applying the principles of sequential designs to meta-analysis have already been developed and could be applied in this context.⁴ An understanding of the clinical issues that need to be addressed can be developed from early studies (phase 1, phase 2, and early phase 3 studies). As a simple example, by regulation, sex and age (adult vs pediatric) must be addressed for a new analgesic. In addition, it would be interesting to consider indication (emergency department, postoperative) and dose (cumulative dose, daily dose, need for a loading dose).

The important principles to keep in mind are planning and controlling variation in the different factors in a systematic manner. We need to be planning "meta-experiments" rather than simply performing post hoc meta-analyses, in the same way as we would plan a factorial experiment or a single randomized trial with stratified randomization. In fact, a number of prospective meta-analyses have already been conducted or are planned.^{5 - 8}

The best way to conduct our meta-experiments, from a statistical or design viewpoint, may not be obvious. It seems to us that we should unconfound design factors as much as possible. Using the same simple example, with sex as the issue of concern and all other things being equal, it seems better to do 2 studies, each including both men and women, and to stratify (either in the randomization or post hoc only) by sex, rather than to do 1 study in men and a separate study in women. As a counterexample, consider the Physicians' Health Study, a randomized trial of 325 mg of aspirin given every other day to prevent myocardial infarction in men,⁹ and its female counterpart addressing similar goals among women.¹⁰ (One reason to study men first and separately in this context is their higher risk of events, with the resulting smaller sample size requirements for an adequately powered study of men.) In the female counterpart, the actual study changed (and, with it, a number of unmeasured associated variables) and dose of aspirin changed (at least in absolute magnitude, although it may be similar on the basis of dose per weight). Perhaps this change in dose is the most appropriate approach clinically, but the choice made by the investigators completely confounds sex, absolute dose, and the actual study and will make the reason for any differences in results difficult to isolate.

An important point is that planning meta-experiments does not need to be limited to planning randomized trials. If one were interested, for example, in whether a particular drug causes a particular adverse event, one might plan a series of case-control studies in different populations (or simply with different choices of control group) to understand better the potential sources of bias inherent to the choice of different reference groups. Such planning assumes that no single study can be definitive and that there is value, for planning future research, to understanding how study design factors relate to estimates of effect.

We recognize that planning will not always be possible and that events will not always go according to plan. Unanticipated clinical issues may arise. However, to the extent possible, it seems reasonable to design meta-experiments, rather than merely to react to existing studies.

To be approved, new drugs must be shown to be both efficacious and safe. In practice, establishing efficacy requires randomized clinical trials with clearly defined end points, valid control groups, appropriate methods for data analysis, and sufficient power to detect differences between treatment and control groups. Furthermore, the evidence must be reproducible, which is generally demonstrated by conducting 2 so-called pivotal trials.¹¹

Research synthesis is already used in the current FDA drug approval process.¹² The requirements in the United States for separate "integrated summaries" of safety and efficacy are really seeking a formal research synthesis, although the implementation may not be entirely in parallel with the emerging science. In fact, since about 1982, the safety evaluation of drugs has included pooled analyses.¹³ Pooled data are also used to look for differences in response among demographic subgroups, particularly groups defined by age and sex.¹³ A less quantitative process of research synthesis is also followed by FDA in the safety review of food additives.

In Australia, where demonstration of cost-effectiveness is required for drug approval, review of all data includes looking at pooled estimates or regressions across studies, trying to assess the likely directions and magnitude of any biases. Meta-analysis is used as a basis for estimating effect sizes, with routine checks of literature searches, data extraction, and data analysis (David A. Henry, MB, FRCP, written communication, August 10, 1998).

It is of note that formal meta-analyses are already being voluntarily submitted in support of applications for health claims for foods and for medical devices. For example, an application for approval of a threaded fusion cage for use in posterior lumbar interbody fusion (PLIF) relied on a now-published meta-analysis of previous PLIF studies¹⁴ to provide data on controls (T. C. Lu, MS, written communication, January 23, 1998). Two recent petitions^{15 - 16} included published meta-analyses^{17 - 18} examining the protective effects of soluble fiber from whole oats or from psyllium seed husk on risk for heart disease (Lynn A. Larsen, PhD, written communication, March 23, 1998).

The idea of using only a meta-analysis in the drug approval process may cause some discomfort. It seems prudent to require that at least 1 adequately powered, well-designed study be performed in support of efficacy. In fact, there may be some reluctance (eg, Probstfield and Applegate¹⁹ ) to accept the results of a meta-analysis in lieu of a large "definitive" trial, especially in light of situations in which the results of meta-analyses and large trials have disagreed (discussed below). We contend, however, that at least under some conditions, a meta-analysis could and should be used to provide independent substantiation of efficacy. In at least some situations, the meta-analysis, by substituting for an additional (not-yet-conducted) large trial, could accelerate the approval process.

Does the fact that a meta-analysis is planned affect our interpretation of the findings? There would be less concern that the reason the meta-analysis was performed was the observation that a number of studies had demonstrated positive effects. On the other hand, one could argue that the original studies would not be pursued to begin with if nobody thought the drug would be effective. In that sense, all drug development is guilty of investigation bias, ie, only investigating things that appear promising. Provided all relevant studies are included in the analysis, selection bias should not be a problem. In the following sections, we discuss alternative approaches to using meta-analysis in the regulatory process.

Suppose the sponsor has conducted a series of 2 or more randomized controlled trials pertinent to a particular question. The trials were planned as a series (a meta-experiment), with prospective registration of studies at their inception and an overall goal of addressing the particular question. The summary estimate of benefit, based on individual patient data, reveals a clinically important effect that is statistically significant. Statistical tests reveal little evidence of heterogeneity. Fixed- and random-effects models give similar results, and preplanned exploration of potential sources of heterogeneity reveals no important modifiers of the benefit of treatment. All the elements of a well-performed meta-analysis are present. The key elements here are the randomization and the preplanning of the entire group of studies as a series of related trials, which goes beyond planning each individual trial in isolation from the others or in response to results of the others.

As we argue above, a major advantage of the meta-analysis in this setting, over the single large trial, is that the meta-analysis might, in fact, offer greater generalizability than a large trial, assuming the results are consistent across studies in spite of heterogeneity of trial design and patient populations. Even when results vary across studies, the meta-analysis could also offer important insights that might be gained from the exploration of reasons for heterogeneity of the apparent treatment effect among studies. We discuss these issues further below.

It is also possible, as a slight variation to the above, to build in an adaptive strategy in which some of the details about a subsequent trial (eg, dose or indication) can depend on the results of a previous trial. Such an adaptive strategy, in fact, is likely to be more feasible than the hypothetical situation in which everything is preplanned, simply because there are likely to be so many unknowns that preplanning everything may not be possible. Nevertheless, one could conceive of preplanning the decision points in the course of action, ie, which study to do next, with what design. The choice of design of subsequent studies would be based on the outcomes of studies already completed, with their results perhaps presented with a continuously updated cumulative meta-analysis or, even more appropriately, a cumulative meta-regression^{20 - 21} (with appropriate concern for issues related to multiple statistical testing²² ). The preplanning would apply primarily to setting up the times at which design decisions are made and the possible routes of action.

We have made no explicit assumption in the above argument about the availability of individual-level (raw) data for the meta-analysis. We do assume, however, that the sponsor has access to all the registered studies, that those studies were registered before their results were known, that appropriate intent-to-treat analyses were performed, and that the results are available in sufficient detail to permit all the relevant adjusted and subgroup analyses. In summary, we assume that there is nothing intrinsically better about individual-level analyses, given a high level of quality control over the group data.

The situation here is similar to that just described, except the meta-analysis is not preplanned. There may be subtle variations that fall between the completely planned meta-analysis and the post hoc meta-analysis. For example, the meta-analysis per se might be planned at some point after the inception or even the completion of the component studies but with no control exerted over exactly what studies get done, where they get done, or by whom and with no prospective registration of studies. In most cases, the interpretation of such analyses should be more cautious than when a meta-analysis is preplanned as part of the approval process. Having urged this caution, we note that we could find a number of claims that prospective meta-analysis should be less biased^{5 - 6 ,23} but found only 2 studies that directly compared prospective with retrospective research synthesis.^{24 - 25} One of these²⁵ found less favorable results in prospectively registered trials (including some unpublished results) than in a meta-analysis of published trials. In the prospectively registered studies, the possibility of publication bias had been eliminated (as the trials were registered prior to having knowledge of their results). The other²⁴ found similar results from prospective and retrospective study assembly.

Relative to the completely planned situation, there is increased danger of bias in the selection of studies, for example, in the form of publication bias. It may be true that only studies with statistically significant results have been published, so that an analysis based solely on published studies (even if the sponsor subsequently obtains individual-level data) may be biased. Yet publication bias is only a problem if a number of unpublished studies exist, with findings contradictory to those of the published studies. The underlying concern is that the significant (published) studies actually motivated the sponsor to make the claim, in contrast to an a priori plan to evaluate the claim. Nevertheless, it is also possible to assume that the "bias" was the decision to conduct such studies and that that decision was driven by an understanding of the underlying biology that suggested the likelihood of a benefit. Why would the study be done if there is truly no suspicion that there will be a benefit? This is particularly a problem for new indications rather than for the primary indication for a new drug. The assembly of studies post hoc invites selection bias and requires mechanisms to counter this effect.

We mention these only briefly in the context of safety studies and others that might be performed in support of medical device claims or health claims for foods. With respect to a regulatory decision, one would certainly want to find results that consistently support a claimed association, but consistency alone is not enough. It is certainly possible to have consistent bias, for example, the case of β-carotene in the prevention of cancer. A series of observational studies (see Ziegler et al²⁶ for review) examined the relation between dietary intake of foods rich in β-carotene and the risk of lung cancer. Overall, they showed a fairly consistent benefit of diets rich in β-carotene that subsequent randomized trials of this specific nutrient have failed to confirm.^{27 - 29}

More typically, one would expect a meta-analysis of observational studies to focus on exploration of heterogeneity of findings. This exploration is particularly important for observational studies because of the lack of control (through randomization) of confounding and bias. Stratification according to particular design features should be performed routinely (eg, case-control studies with hospital controls vs case-control studies with community controls vs cohort studies). Other tools, such as regression models, may be used to obtain a best estimate of an association, adjusted for particular elements of study design, or to predict a value of the finding for a particular combination of study attributes. Dose response (or duration response) can be evaluated across studies using meta-analytic techniques,³⁰ either by comparing relative risk estimates among high- and low-dose studies or by using pooled within-study estimates of dose response.^{30 - 32}

Several empirical studies have demonstrated that large trials can disagree with previous (or concurrent) meta-analysis.^{33 - 40} The first of the analyses cited³³ assumes that the large randomized trials are correct. For several reasons, the idea that bigger is better is not a premise we accept. First, replication of a finding by independent studies should surely provide stronger evidence than a single trial, and second, large trials can produce different answers because of important differences in study design. For BCG vaccine, a huge trial using passive follow-up and therefore missing at least 50% of all cases of tuberculosis failed to show the protective effect demonstrated in numerous other, more carefully conducted trials and case-control evaluations of BCG vaccine.^{41 - 42} Several other explanations for discrepancies between large trials and meta-analyses have been suggested, including publication bias,⁴³ difficulties in implementation of large trials,⁴⁴ and null bias caused by contamination of control groups with patients using nontrial therapies.⁴⁵ In specific cases, differences in patient populations or protocols^{46 - 47} have been hypothesized to lead to differences in study results.

While it is not our purpose to explore in detail discrepancies between large trials and meta-analyses, it remains unclear whether the assumption is correct that the meta-analysis is wrong and the large trial is right. One would certainly question whether to believe a large trial or pair of trials showing efficacy, in the face of a number of well-designed smaller trials with point estimates of treatment effect at or near the null, except if the large trials demonstrated that they specifically addressed populations for whom treatment is uniquely effective. A certain amount of variability of results is to be expected across studies. Understanding the clinical reasons for such variability and discrepancies in results and the potential sources of bias in the studies becomes the challenge of the approval decision. These issues can be effectively addressed through meta-analysis.

Through this review we have considered the potential applications of meta-analysis and related principles of replication of studies in the ongoing research synthesis required for regulatory approval. Table 1 summarizes the strengths of different strategies that might be used in the regulatory process. The primary goals of more widespread use of meta-analysis in the approval process are the potential streamlining of that process and the quantification and understanding of heterogeneity of treatment effects across studies. Appropriately designed and conducted meta-analyses may achieve rapid demonstration of effectiveness in at least 1 relevant subpopulation, with the goal of getting a product onto the market that will benefit (and be safe in) at least some patients (preferably those most in need of therapeutic benefit). A separate issue that we do not address is limiting subsequent use only to those patients in whom effectiveness is demonstrated.

A second goal of meta-analysis is to identify important sources of variability in effectiveness (at least on an exploratory basis). Where possible or necessary, meta-analysis or, more properly, meta-experiments and principles of experimental design can be used to exploit those sources of variability in the design of subsequent trials and ultimately in guiding clinical practice. Clearly, there is more work to be done before we can achieve this goal.

In summary, what we are looking for from the meta-analysis is support of claims. This support, in the form of a quantitative summary, should not replace the traditional review or the integrated summary but should supplement it. That is, the meta-analysis should not substitute a mechanical application of statistical techniques for careful, considered judgment but should use quantitative methods to explore sources and magnitude of heterogeneity and bias.