Stopping Randomized Trials Early for Benefit and Estimation of Treatment Effects:  Systematic Review and Meta-regression Analysis FREE

Although randomized controlled trials (RCTs) generally provide credible evidence of treatment effects, multiple problems may emerge when investigators terminate a trial earlier than planned,¹ especially when the decision to terminate the trial is based on the finding of an apparently beneficial treatment effect. Bias may arise because large random fluctuations of the estimated treatment effect can occur, particularly early in the progress of a trial.² When investigators stop a trial based on an apparently beneficial treatment effect, their results may therefore provide misleading estimates of the benefit.^3- 4 Statistical modeling suggests that RCTs stopped early for benefit (truncated RCTs) will systematically overestimate treatment effects,⁵ and empirical data demonstrate that truncated RCTs often show implausibly large treatment effects.⁶

Empirical evidence addressing the magnitude of bias from stopping early, and factors that may influence the magnitude of the bias, remain limited and the appropriate interpretation of truncated RCTs a matter of controversy.^6- 11 We therefore undertook a systematic review to determine the treatment effect from truncated RCTs compared with meta-analyses of RCTs addressing the same research question that were not stopped early (nontruncated RCTs) and to explore factors associated with overestimates of effect.

A prior report provides a detailed description of the design and methods of this study (Study of Trial Policy of Interim Truncation-2 [STOPIT-2]).¹² In summary, we conducted extensive literature searches to identify truncated RCTs and systematic reviews addressing the same question. We retrieved all RCTs included in the systematic reviews, extracted data and conducted new meta-analyses of the nontruncated RCTs addressing the outcome that led to the early termination of the truncated RCTs, and compared the relative risk (RR) generated by the truncated RCTs with the RR from all matching nontruncated RCTs.

We updated the database from our prior study following the same search strategy.⁶ In January 2007 we searched MEDLINE, EMBASE, Current Contents, and full-text journal content databases from their inception for truncated RCTs. In addition, we identified truncated RCTs through hand searching, by personal contact with trial investigators, and by a citation search linked to 2 key articles.^6,13 For systematic reviews, we searched the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects, and MEDLINE from their inception to January 2008.

We included RCTs of any intervention reported as having stopped earlier than initially planned owing to interim results in favor of the intervention.

We excluded matching systematic reviews that did not have a methods section and did not describe a literature search that, at minimum, included MEDLINE.¹²

We retrieved the full text of all RCTs included in each systematic review. If a systematic review was published prior to the matching truncated RCT and thus did not include the truncated RCT, we updated this review.¹² Eligible nontruncated RCTs addressed the outcome that led to the early termination of the truncated RCT and stated clearly that allocation was randomized. We assessed the eligibility of nontruncated RCTs based on the similarity of the question addressed by the matching truncated RCT (see Briel et al¹² for details).

Teams of 2 reviewers with relevant clinical expertise made independent eligibility and similarity decisions and resolved disagreement by discussion and, if necessary, by consulting a third party. Reviewers who judged eligibility were blinded to the results of the trials through electronic or manual masking.¹²

Working in pairs, reviewers with methodological expertise conducted data extraction independently.¹² From each RCT (truncated or nontruncated), we collected information about early termination, the journal of publication (we categorized Annals of Internal Medicine, BMJ, JAMA, Lancet, and New England Journal of Medicine as high-impact journals), the year of publication, methodological quality, data monitoring committees, stopping rules at the outset of the trial, interim analyses, and the measure of treatment effect for the outcome that terminated the truncated RCT. The only study characteristic tested for a statistically significant difference between truncated and nontruncated RCTs was publication in a high-impact journal.

We calculated an RR for each RCT in our study. For studies that provided results using continuous data, we estimated an approximate dichotomous equivalent.^12,14 For each question, we used meta-analysis for the pooled RR and a 95% confidence interval (CI) for all nontruncated RCTs. If more than 1 truncated RCT addressed the same question, we calculated a pooled RR and CI for those truncated RCTs. Pooled estimates of RRs were calculated using an inverse-variance weighted random-effects model.

We performed a z test for each meta-analysis to assess differences between the truncated and nontruncated RCTs with respect to their pooled RRs. As a summary measure we calculated a ratio of RRs, and its logarithm, for each question as follows:

Log[ratio of RRs] = log[RR of truncated RCT(s)/pooled RR of nontruncated RCTs] = log[RR of truncated RCT(s)] – log[pooled RR of nontruncated RCTs].

We estimated the pooled log[ratio of RRs] using a random-effects inverse-variance meta-analysis and then, for purposes of presentation, back transformed to the overall ratio of RRs. To explore factors associated with the magnitude of the ratio of RRs, we performed a meta-regression analysis in which the dependent variable was the log[ratio of RRs] and independent variables were whether the truncated RCTs used a formal stopping rule and the number of outcome events in the truncated RCTs. When more than 1 truncated RCT addressed the same question, the stopping rule status was assigned to “has a rule” if at least 1 truncated RCT had a rule. Similarly, when there was more than 1 truncated RCT for the same question, we used the truncated RCT with the largest number of events as the source for our analyses of the influence of the number of events.

To allow consideration of methodological quality as a predictor and to test whether restriction to nontruncated RCTs that are more similar to the truncated RCTs would change the results, we constructed a second meta-regression described fully in a prior report.¹² In brief, this meta-regression used a hierarchical model with 2 levels: individual RCT (study) level and meta-analysis (question) level. The dependent variable in this analysis was the logarithm of the RR for each study. Predictor variables considered included a combined group variable (truncated RCT with a rule, truncated RCT without a rule, nontruncated RCT), number of events, concealment of allocation, use of blinding, and the interaction between the group variable and the other variables. We performed this meta-regression on different data sets based on various thresholds for the similarity of the nontruncated RCTs in each question to the matching truncated RCTs.

To test for an order effect (the hypothesis being that studies published earlier will have more responsive populations), for each review question we established where in the sequence of published studies (by publication date) the truncated RCT stood and referred to this as the “rank” of the truncated RCT. We then calculated a “standardized rank” [100 · (rank − 1)/(total number of studies − 1)]. If there was more than 1 truncated RCT in the review question, we used the median among the truncated RCTs as the standardized rank of truncated RCTs. We then divided the review questions into 2 groups (those with the standardized rank of the truncated RCT equal to or less than 50 [n = 27] and those with the standardized rank of the truncated RCT greater than 50 [n = 37]) and repeated the meta-analysis for each group.

As a secondary analysis, we compared the RR of the truncated RCT(s) with the pooled estimate for all trials including the truncated RCTs.

Analyses were performed using SAS version 9.2 (SAS Institute Inc, Cary, North Carolina); tests were 2-sided, and P < .05 was used as the threshold for statistical significance.

A total of 195 truncated RCTs formed the basis for the search for systematic reviews; we identified matching systematic reviews for 79 questions. We extracted 2488 nontruncated RCTs from 202 matching systematic reviews (of which 32 were updated). Of these 2488 studies, 22 (0.9%) proved to be truncated RCTs, which we added to the truncated RCT database. We excluded 2012 nontruncated RCTs based on insufficient similarity to the truncated RCTs or unclear randomization and 30 because the RR could not be calculated. The remaining 424 nontruncated RCTs and 91 matching truncated RCTs addressed 63 questions (Figure 1). An eSupplement reporting the references of the included studies is available.

Table 1 describes the characteristics of the eligible studies. Compared with matching nontruncated RCTs, truncated RCTs were more likely to be published in high-impact journals (30% vs 68%, P < .001).

Of 63 comparisons, the ratio of RRs was equal to or less than 1.0 in 55 (87%); the weighted average ratio of RRs was 0.71 (95% CI, 0.65-0.77; P < .001) (Figure 2). In 39 of 63 comparisons (62%), the pooled estimates for nontruncated RCTs were not statistically significant.

Comparison of the truncated RCTs with all RCTs (including the truncated RCTs) demonstrated a weighted average ratio of RRs of 0.85; in 16 of 63 comparisons (25%), the pooled estimate failed to demonstrate a significant effect.

Table 2 summarizes the findings from the single-level meta-regression analysis to determine predictors of differences in the treatment effect size between truncated and nontruncated RCTs. In the univariable models, both the number of events (P < .001) and the presence of a statistical stopping rule (P = .02) were significant. When we included both variables in the model, only the number of events remained significant (P < .001). The results from the multilevel meta-regression confirmed significant interactions between the combined variable (truncated vs nontruncated RCT) and the number of events (P < .001). Large differences in treatment effect size between truncated and nontruncated RCTs (ratio of RRs <0.75) occurred in truncated RCTs with fewer than 500 events (Figure 3).

The multilevel meta-regression analysis using the entire data set demonstrated that neither concealment of allocation (P = .96) nor blinding (P = .32) were significant predictors of the differences in treatment effect size.

The findings were similar, irrespective of either the closeness of the match between nontruncated and truncated RCTs or the order of publication of the truncated RCTs relative to that of matching nontruncated RCTs. In the multilevel meta-regression analysis, adjusted ratios of RRs of truncated vs nontruncated RCTs were 0.64 when questions were very closely matched, 0.70 when they were moderately close, and 0.69 when they were least close. The ratio of RRs of the group in which the truncated RCTs were published in early years (standardized rank ≤50) was 0.74 (95% CI, 0.66-0.83) and for the later years (standardized rank >50) was 0.68 (95% CI, 0.60-0.77). The P value for the difference between the 2 estimates was 0.33.

In this empirical study including 91 truncated RCTs and 424 matching nontruncated RCTs addressing 63 questions, we found that truncated RCTs provide biased estimates of effects on the outcome that precipitated early stopping. On average, the ratio of RRs in the truncated RCTs and matching nontruncated RCTs was 0.71. This implies that, for instance, if the RR from the nontruncated RCTs was 0.8 (a 20% relative risk reduction), the RR from the truncated RCTs would be on average approximately 0.57 (a 43% relative risk reduction, more than double the estimate of benefit). Nontruncated RCTs with no evidence of benefit—ie, with an RR of 1.0—would on average be associated with a 29% relative risk reduction in truncated RCTs addressing the same question.

In nearly two-thirds of comparisons, the pooled estimate for nontruncated RCTs failed to demonstrate a statistically significant effect. We found substantial heterogeneity in our analysis of the pooled ratio of RRs for truncated vs nontruncated RCTs, suggesting that differences between truncated and nontruncated RCT effects will differ across study questions. This heterogeneity could be partially explained by the total number of outcome events in the truncated RCTs, with larger differences between truncated and nontruncated RCTs in studies with a smaller number of events.

The methodological quality and the presence of a statistical stopping rule failed to predict the observed difference in the treatment effect.

We used rigorous search strategies and undertook an intensive independent evaluation of eligibility and similarity of several thousand RCTs blinded to the results. Our analysis had considerable statistical power to link the estimates of treatment effect from truncated and nontruncated RCTs addressing the same question and demonstrated consistent results across degrees of similarity of the question addressed by the truncated RCTs and the matching nontruncated RCTs.

Our literature search, while extensive, missed some truncated RCTs. Assessment of the 2488 RCTs included in the systematic reviews revealed 22 additional truncated RCTs not initially identified. Whether results would differ in other unidentified truncated RCTs remains speculative.

We relied on systematic reviews to identify nontruncated RCTs but did not assess the reviews' susceptibility to publication bias. However, we know that trials with positive findings have nearly 4 times the odds of being published compared with those with negative findings.¹⁵ To the extent that publication bias is present, inclusion of unpublished studies would lead to a diminished pooled effect from the nontruncated RCTs. This would in turn likely lead to a larger gradient of effect between truncated and nontruncated RCTs. Thus, to the extent that publication bias exists, our results probably represent a conservative estimate of the exaggeration in treatment benefit associated with stopping early.

Korn and colleagues recently reviewed the results of cancer trials stopped early and that either continued with further follow-up or released results early.¹⁶ They found that substantial differences between results at the time of early stopping and subsequent follow-up seldom occurred.¹⁶ Freidlin and Korn published a related simulation study that supported these findings, suggesting that if the true effect is large, differences between stopped-early results and full follow-up results will differ little.¹⁷

These recent studies confirm that even when the true effect is large, studies stopped early still overestimate that effect. More important, the authors do not address circumstances in which the true underlying effect is small or absent. Clinicians seek the best estimate of an unknown true underlying effect with appropriate safeguards against bias. As Goodman¹⁸ points out in a commentary on the simulation by Freidlin and Korn, “since we do not know what the true effect is, we cannot know in any particular case whether the observed effect is biased or not; the fact that the trial is stopped early is not prima facie evidence that the estimate is wrong.” We support this statement; unfortunately, neither do we know that the stopped-early result is close to the truth. Our findings suggest that often it is not.

Consensus exists that rigorous data monitoring practice requires a predefined statistical stopping rule.^19- 20 Our findings, however, indicate that even a formal rule is insufficient to prevent bias consequent on stopping early and suggest the advisability of rules that require a large number of outcome events before early stopping is contemplated.

In this review we have focused only on RCTs stopped early for benefit. Although ethical concerns make decisions regarding stopping trials early for safety more complex than those regarding stopping trials early for benefit, inferences regarding harm and those regarding benefit are equally susceptible to the bias associated with stopping early.

Our results have important implications for systematic reviews and ethics.^21- 22 If reviewers do not note truncation and do not consider early stopping for benefit, meta-analyses will report overestimates of effects.²¹ Investigators and funding bodies—in particular, drug and device manufacturers—have different but convergent interests to stop a study as soon as an important difference between experimental and control groups emerges, and journals have an interest in publishing the apparently exciting findings. Furthermore, data monitoring committees are well aware of their ethical obligation to ensure that patients are offered effective treatment as soon as it is clear that effective treatment is indeed available, providing a justification for stopping early.

However, data monitoring committees also have an ethical obligation to future patients who need to know more than whether data crossed a significance threshold; these patients need precise and accurate data on patient-important outcomes, of both risk and benefits, to make treatment choices.²² Such patients will often number in the tens or hundreds of thousands and sometimes in the millions. To the extent that substantial overestimates of treatment effect are widely disseminated, patients and clinicians will be misled when trying to balance benefits, harms, inconvenience, and cost of a possible health care intervention. If the true treatment effect is negligible or absent—as our results suggest it sometimes might be—acting on the results of a trial stopped early will be even more problematic. Thus, for trial investigators, our results suggest the desirability of stopping rules demanding large numbers of events. For clinicians, they suggest the necessity of assuming the likelihood of appreciable overestimates of effect in trials stopped early.