Resolving Discrepancies Between a Meta-analysis and a Subsequent Large Controlled Trial FREE

Meta-analysis has become increasingly popular in medical research in which information about treatment efficacy is available from a number of clinical trials with inconclusive or inconsistent results.^1- 3 While such analyses are often useful for summarizing the evidence and identifying sources of heterogeneity, their value in assessing the efficacy of a clinical intervention remains controversial.^4- 6 Increasing reports of discrepancies between the results of meta-analyses and those of subsequent large randomized clinical trials^7- 10 have reinforced the debate and underscored the need for further evaluation of the current practice of meta-analysis in medical research.^11- 13

The results of a recent meta-analysis¹⁴ of calcium supplementation for the prevention of preeclampsia and those of a large trial subsequently published^15- 16 illustrate the large discrepancies that may be encountered. The meta-analysis found calcium to be highly effective in preventing preeclampsia with an overall odds ratio (OR) of 0.38 and a corresponding 95% confidence interval (CI) from 0.22 to 0.65.¹⁴ In contrast, results from the National Institutes of Health Calcium for Preeclampsia Prevention (CPEP) trial indicated no reduction of preeclampsia risk in healthy nulliparous women due to calcium supplementation (relative risk [RR], 0.94; 95% CI, 0.76-1.16).¹⁵

The trials included in the meta-analysis were heterogeneous in several respects. Whereas some included only healthy nulliparous women without additional known risk factors for preeclampsia, others enrolled exclusively women identified through clinical testing to be at high risk of hypertensive disorders. Many did not include a placebo-controlled group and thus were unable to ensure that subjects and investigators were blinded to the treatment assignment. A careful review of end point definitions in the individual trials is essential to assure that the meta-analysis includes only results for preeclampsia. Preeclampsia is characterized by the de novo onset of elevated blood pressure and proteinuria after 20 weeks of gestation. The published meta-analysis included 1 trial that reported data only for gestational hypertension without proteinuria,¹⁷ and another that failed to define preeclampsia,¹⁸ so it was impossible to verify that the end point studied was indeed preeclampsia and not gestational hypertension.

Using all available trials of calcium supplementation for the prevention of preeclampsia published prior to the CPEP trial, we have reanalyzed the data and focused attention on issues relevant to effect heterogeneity and understanding its sources. Such issues are often overlooked in the current practice of meta-analysis, yet they must be addressed before it is possible to draw proper inferences about treatment efficacy.^19- 21

Through searches of the MEDLINE bibliographic database and references in the articles selected, we identified 14 trials^{15,18,22- 33} published by July 10, 1997, the date of publication of the major results of the CPEP trial.¹⁵ All 14 trials reported outcome data on preeclampsia and a treatment regimen including calcium supplementation. One trial³⁰ had also been reported in 2 separate abstracts^34- 35 and subsets of the data of another¹⁸ had appeared in 2 additional articles.^36- 37 For these 2 trials we considered only the information from the most comprehensive report.^18,30 To focus on the CPEP trial and the trials leading up to it, our database excludes a trial³⁸ reported after publication of the major results from the CPEP trial.¹⁵

We recorded the sample size and number of subjects who developed preeclampsia for each treatment group in each trial. Each author independently extracted the data. To ensure uniformity of follow-up data and to minimize bias, all subjects were included in the treatment group to which they had been originally assigned, regardless of follow-up experience.³⁹ For instance, in 1 trial²² we included all 106 subjects enrolled despite the fact that the authors had eliminated 14 women from the analysis "because they were delivered before 38 weeks, although ... none had any signs of PIH [pregnancy-induced hypertension]."

The RRs with corresponding 95% CIs were computed for each trial using the logit model and pooled sampling variances based on the overall number of preeclampsia events in the 2 treatment groups. Effect homogeneity was evaluated by the Q statistic⁴⁰ using the RR as a measure of calcium efficacy. Random-effects model estimates and 99% CIs for the summary RRs were estimated according to DerSimonian and Laird.⁴⁰ To mitigate the impact of cells with zero or small counts on the overall conclusions, tests of homogeneity and summary estimates were based on weights reflecting the individual pooled sampling variances.⁴¹

Characteristics of the 14 trials, which could influence underlying control rates of preeclampsia or the effects of calcium supplementation, are summarized in Table 1 and Table 2. In addition to baseline and design attributes, such as parity, gestational age, dietary intake, and treatment dose, these 2 tables include items that give a qualitative assessment of the methodological strengths or weaknesses of each trial, which are random treatment assignment, inclusion of placebo-control subject, and sample size. In the 6 trials that were placebo-controlled and randomized (Table 1),^15,22- 26 both study subjects and investigators were blinded to treatment assignment. Total sample size ranged from 56 to 4589. Calcium therapy in all 6 trials was 2000 mg/d, but reported average dietary calcium intake ranged from 300 to 1200 mg/d, reflecting the dietary diversity of the subject populations.

Although these trials almost always enrolled nulliparous women, 2 trials^23,26 selected only women at high risk of preeclampsia, identified either by means of a positive rollover test²³ or a positive rollover test followed by a positive angiotensin-sensitivity test.²⁶ A rollover test was considered positive if the diastolic blood pressure increased to 20 mm Hg or more when the patient was turned from the left-lateral recumbent to the supine position. The angiotensin-sensitivity test measures the dose of infused angiotensin II required to elicit a 20-mm Hg increase in diastolic blood pressure. A positive angiotensin-sensitivity test was an effective pressor dose of less than 12 ng/kg per minute. Either a positive rollover or a positive angiotensin-sensitivity test result has been associated with a 2-fold to 3-fold elevation in risk of the hypertensive disorders of pregnancy.

The 2 trials of women identified in this way to be at high risk of preeclampsia, as expected, reported high rates of preeclampsia in the placebo group (23.5% and 44.1%). The authors of the trial in women with positive rollover test results²³ reported a high rate of preeclampsia (27.9%) in the placebo group of another trial,²² although that report did not indicate selection of high-risk subjects. In this trial, surprisingly, all women with pregnancy-induced hypertension were noted to be proteinuric.²² Moreover, the definition of hypertension allowed for a 30-mm Hg systolic or a 15-mm Hg diastolic increase in blood pressure and did not require blood pressure elevation to the 140/90 threshold. All 3 trials with high placebo rates^22- 23,26 had enrolled healthy subjects with relatively low dietary calcium intake (300-600 mg/d).

Table 2 summarizes the 8 trials that did not include a placebo-controlled group.^18,27- 33 The trials ranged in size from 100 to 3042 and preeclampsia control rates varied from 2.0% to 17.0%. Four of the 8 trials without placebo-controlled subjects did not specify whether assignment of subjects to treatment or control groups had been random, and most did not report dietary calcium intake. In 5 trials,^{27- 29,31- 32} the treatment group had received a therapeutic regimen that included other minerals and vitamins in addition to relatively little supplemental calcium (<400 mg/d). A dose of 2000 mg/d, similar to that of all placebo-controlled trials, had been administered in only 2 of the trials^18,30 without placebo-controlled groups.

Table 3 lists numbers of subjects, numbers who developed preeclampsia, and corresponding RR estimates and 95% CIs for each of the 14 trials by treatment group. All 14 trials reported a beneficial effect of calcium on preeclampsia with observed risk ratios ranging from 0.09 to 0.94 (Table 3). CPEP,¹⁵ the largest trial and the only one designed with adequate power to test efficacy for preeclampsia in healthy nulliparous women, indicated the least benefit (RR, 0.94) and did not achieve statistical significance. The trial by Lopez-Jaramillo et al²³ had the smallest sample size but reported the greatest benefit of calcium and attained statistical significance. In all, 3 trials^22- 23,26 with and 5 trials^{18,28- 30,32} without placebo-controlled subjects reported statistically significant reductions in risk.

Table 4 identifies sources of the observed effect heterogeneity and summarizes the overall results within each of the homogeneous subgroups. Overall effect estimates and 99% CIs are based on the random-effects model. When true treatment is constant across all trials, fixed and random models yield similar results. On the other hand, when treatment effect varies across the trials, only results based on the random model are generalizable to settings similar to those in the analysis.⁴²

Because individual rates were available from all trials, we used the risk ratio as a measure of RR since it is simpler and easier to understand than the OR. To confirm the robustness of our results to the choice of treatment effect measure and to the zero and/or small counts in the data, we included the OR in several sensitivity analyses. When statistical heterogeneity was not observed, we also considered the fixed-effects model to assess the consistency of the overall conclusions to the choice of the estimation method.

Table 4 indicates enormous effect heterogeneity (P=.001) across the 14 trials, reflecting wide differences in study design and observed effects. This heterogeneity cannot be attributed solely to the CPEP trial since it was also observed (P=.005) in data published prior to the CPEP trial.¹⁵ Stratifying the data from trials with and without a placebo-controlled group reduced some, but not all, of the observed variation of effects. Whereas data from the 8 trials without a placebo group were statistically homogeneous (P=.15), effect heterogeneity persisted in the placebo-controlled trials^15,22- 26 with (P=.001) or without (P=.03). The assumption of a constant RR across the trials, however, held (P=.25) within each subset of high-risk^22- 23,26 or low-risk^15,24- 25 populations. Stratification into low-risk and high-risk subgroups was based on a priori considerations that took into account differences in subject selection. We included the trial by Lopez-Jaramillo et al²² in the high-risk set since this trial reported a rate of preeclampsia (27.9%) in the placebo group similar to that of the 2 placebo-controlled trials,^23,26 which had enrolled only high-risk women identified through clinical testing. When including the trial by Lopez-Jaramillo et al,²² the low-risk group yielded enormous heterogeneity (P=.005), but did not affect conclusions about calcium efficacy.

Individual RR estimates symmetrical in the logarithmic scale and 95% CIs are depicted for the 6 placebo-controlled trials in Figure 1 and for the 8 trials without a placebo-controlled group in Figure 2. Overall RR estimates and corresponding 99% CIs are included for subsets of trials with statistical homogeneity. For the trials in low-risk populations^15,24- 25 with a total sample size of 5946 and 7% preeclampsia risk among the placebo-controlled subjects, the summary RR estimate was 0.79 (99% CI, 0.44-1.42; P=.31). Corresponding fixed model estimates (RR, 0.90; 99% CI, 0.69-1.16), strongly reflecting CPEP data, also implied a null calcium effect. After excluding the CPEP trial, the data were also consistent with the null hypothesis of no benefit due to calcium. The 1357 women in the other 2 trials with low-risk populations^24- 25 had a preeclampsia risk of 3.8% among the placebo-controlled subjects and an overall RR of 0.52 (99% CI, 0.13-2.02; P=.22). The 225 women in the high-risk group^22- 23,26 with 29% preeclampsia among placebo-controlled subjects had an overall RR of 0.19 (99% CI, 0.08-0.46; P=.001).

Statistical heterogeneity of effect was not detected among the 8 trials without placebo groups (P=.15) and overall estimates indicated a beneficial calcium effect (P=.001). With a total sample size of 5839 and a mean control rate of 9%, these 8 trials yielded an overall RR of 0.53 (99% CI, 0.36-0.79).

Our results indicate that if the enormous heterogeneity effects were ignored, overall results with or without CPEP data would imply a strong beneficial calcium effect. This inference would agree completely with that of the recent meta-analysis, despite its use of different criteria for inclusion of trials, different criteria for data collection, and different methods for estimation of effects. For instance, the recent meta-analysis¹⁴ included 9 trials published before CPEP. Except for CPEP, the meta-analysis¹⁴ considered all 5 placebo-controlled trials in Table 1 and 3 trials^18,30- 31 without a placebo group in Table 2. The 3 trials without a placebo group^18,30- 31 that had been included in the meta-analysis were evaluated with the placebo-controlled trials. Of the remaining 5 trials in Table 2 without a placebo-controlled group, 2 trials^32- 33 were published subsequent to the meta-analysis and 3 trials^27- 29 were excluded.

There are other differences in data and estimation methods between our analysis and the previous meta-analysis. For the trial by Cong et al,¹⁸ the meta-analysis considered the data from the pilot study with a sample size of 100, whereas we used the data from the main study with a sample size of 400. While we used all subjects enrolled, the meta-analysis sometimes excluded subjects lost to follow-up. The meta-analysis reported overall effect estimates using a fixed model in the absence of statistically significant heterogeneity and a random model in the presence of significant heterogeneity. Significant heterogeneity was not observed in the studies with preeclampsia.

Even with these differences, however, the results of the meta-analysis of preeclampsia correspond well with the combined results of Table 4. The meta-analysis reported an overall OR of 0.38 (95% CI, 0.22-0.65), which is similar to the overall estimates for all trials with or without data from the CPEP trial (RR, 0.45 [95% CI, 0.29-0.69] and RR, 0.41 [95% CI, 0.27-0.64], respectively).

Our evaluation of the previous data indicates that results from CPEP are consistent with those from other placebo-controlled trials in healthy nulliparous women not selected for high risk. CPEP, a large placebo-controlled multicenter trial,¹⁶ did not find evidence of a beneficial effect in such women due to calcium supplementation. Sensitivity analyses indicate that conclusions about overall efficacy and identification of covariates, which explain effect heterogeneity, remain robust to some differences in data and methods. Whether we include or exclude CPEP data, use a random or a fixed-effects model, include or exclude subjects lost to follow-up, use the RR or the OR as the measure of treatment effect, the summary results remain consistent with the null hypothesis of no reduction in preeclampsia risk due to calcium supplementation in healthy low-risk populations.

For healthy high-risk subject populations, however, our summary results imply a beneficial calcium effect. Each of the 3 placebo-controlled trials that assessed the efficacy of calcium in high-risk populations reported a large and statistically significant reduction in preeclampsia risk. Overall, the homogeneous observed effects implied about an 80% reduction in risk due to calcium supplementation. A biological rationale for the differences in effect estimates between the high-risk and low-risk groups could be attributed to differences in dietary calcium intake,⁴³ which ranged between 300 to 600 mg/d and 600 to 1200 mg/d in the high-risk and low-risk groups, respectively (Table 1). Nevertheless, the significance of the data from the 3 trials with high-risk subject populations remains questionable in light of the small overall sample size. The following facts remain to be explained: (1) 2 trials^22- 23 had been conducted by the same investigators among women living at high altitude in the vicinity of Quito, Ecuador, (2) data from 1 trial had been reported in a letter to the editor²³ and not in a peer-reviewed article, and (3) the inclusion of a trial with a high placebo rate of preeclampsia in an unselected population and the absence of gestational hypertension without proteinuria.²² Additional carefully controlled randomized trials are needed to establish whether calcium supplementation may indeed reduce the incidence of preeclampsia in healthy women at high risk, especially in the presence of low dietary calcium intake.

The validity of the results from the 8 trials without a placebo group is dubious. The women in the control groups of these trials were followed up for preeclampsia, but did not receive any study medication. In the absence of a placebo-controlled group, any effect of calcium supplementation cannot be distinguished from a placebo effect. Moreover, lack of a placebo group often precludes blinding of subjects and investigators and increases the potential for bias in assessing calcium efficacy.^44- 45 Furthermore, 5 of the 8 trials evaluated only the effects of a relatively small amount of calcium, less than 400 mg/d, given with other vitamins and minerals. A beneficial effect in these studies may reflect the efficacy of other components of the therapeutic regimen besides calcium.

An important limitation of our study should be mentioned. Although our subgroup analyses are based on potential covariates considered a priori, they do not reflect a priori hypotheses based on biological plausibility.⁴⁶ To reduce the possibility of finding spurious associations, it is preferable to perform analyses of biologically plausible subgroups.⁴⁶ In practice, however, this may not be possible since pertinent covariate information is not always available. In the trials of calcium for preeclampsia prevention, for example, information about dietary calcium intake is missing from all but 1 of the trials without a placebo-controlled group (Table 2). When biologically plausible variables are not readily available to explain observed heterogeneity, simple surrogate measures that assess the quality of study design (eg, use of placebo-controlled subjects) or summarize important, but unknown, patient risk factors (eg, placebo-controlled rate) provide valuable assistance in understanding the data and in interpreting the results.⁴⁷

The most notable finding to emerge from our investigation is the fact that a random model cannot cure all levels of heterogeneity, especially when underlying heterogeneity is not minimal. In the absence of statistically significant heterogeneity, the random model may be preferable to the fixed-effects model since it yields a more conservative assessment of overall efficacy and the results are more readily generalized. Moreover, use of the random model may be desirable in view of the generally poor power of heterogeneity tests and the fact that statistical homogeneity does not demonstrate clinical homogeneity. In the presence of statistically significant heterogeneity, however, a single summary measure even from the random model cannot adequately describe the data.^19- 21,47 In such settings, the primary objective should be to explore the reasons for the observed effect heterogeneity. Incorporating pertinent covariate information into the analysis of treatment efficacy may reduce the underlying effect heterogeneity. The summary estimates will then be based on homogeneous populations and allow for more specific therapeutic recommendations.