Quality of Reporting of Noninferiority and Equivalence Randomized Trials FREE

A noninferiority or equivalence trial aims to demonstrate that the experimental treatment is not clinically worse than the comparator (an active control treatment) by more than a prespecified small amount (Δ), known as the noninferiority or equivalence margin.¹ According to the International Conference on Harmonization guidelines,² the term noninferiority is used when referring to a 1-sided trial (difference in response lower than Δ); equivalence, when referring to 2-sided trials (difference in response between −Δ and +Δ). The new treatment is expected to have noninferior or equivalent efficacy compared with the standard treatment but could have advantages in safety, convenience (eg, administration once a day instead of 3 times a day), or cost. The new treatment may also present an alternative or second-line therapy.³

Noninferiority or equivalence trials imply particular planning and analysis.^4- 9 The noninferiority margin is taken into account in the formulation of the sample size calculation.^10- 11 The margin must be smaller than or equal to “the smallest value that would represent a clinically meaningful difference, or the largest value that would represent a clinically meaningless difference.”¹² The determination of this margin must be based on both statistical reasoning and clinical judgment.¹³ For analysis, intent-to-treat (ITT) and per-protocol analyses should be performed.^2,5,14 Results are usually displayed as confidence intervals (CIs) around the observed differences in responses, and conclusions are then drawn by comparing these CIs with the prespecified noninferiority margin.³

An important review of reports claiming equivalence published between 1992 and 1996,¹⁵ assessing whether these claims were consistent with methods and results, concluded that about half were not aimed at demonstrating equivalence. We sought a new study with a different mode of selecting trials (trials aiming to investigate either noninferiority or equivalence instead of trials claiming equivalence) from a sample of recent trials conducted after the publication of guidance from regulatory authorities. Therefore, we systematically reviewed noninferiority and equivalence randomized controlled trials published in 2003 and 2004, focusing on methodologic elements specific to such trials. We aimed to highlight the areas about which misconceptions and problems in the reporting of noninferiority or equivalence trials prevail.

We performed a computerized search of the MEDLINE databases and the Cochrane Central Register of Controlled Trials using the search terms equivalenceORequivalentORnoninferiorityORnoninferior. We identified all reports of clinical trials published in the English language from January 1, 2003, to December 31, 2004. One of us (A.L.H.) then screened the titles and abstracts to identify potentially relevant studies, and the final selection was made from reading the full text. Articles were included only if the study was identified as a randomized controlled trial assessing noninferiority or equivalence (excluding bioequivalence studies). Letters and articles for which only the abstract was available or reports describing only the design of the trial were excluded. Articles were screened for duplicate publication (ie, the same trial described in several articles), and only the article presenting the main results was selected.

Two independent reviewers (A.L.H. and G.B.) tested a data extraction form with a distinct set of 10 articles during a training session. Then, during a meeting, they discussed the interpretation of the different items. To assess interobserver reliability, a subsample of 30 articles was randomly selected from the sample of included articles, and the 2 reviewers independently extracted information. For the remaining articles, a single reviewer (A.L.H.) extracted information. Reviewers were not blinded to the journal name and authors.

The following data were extracted from all reports:

Categorical variables were described with frequencies and percentages. The degree of agreement between the 2 reviewers was determined with use of the κ coefficient. All data analyses involved use of SAS version 9.1 (SAS Institute Inc, Cary, NC).

A flowchart of the selection of reports is presented in the Figure. The electronic search yielded 1191 citations. From this list, 249 potentially relevant articles were selected after screening titles and abstracts, and, finally, 162 reports were selected after full-text reading. We thus reviewed 116 (71.6%) noninferiority trials and 46 (28.4%) equivalence trials.

The κ coefficients varied between 0.42 and 1.00. The lowest κ was observed for the item that assessed whether the conclusions of the article agreed with the results, but it was associated with a 74% concordance rate.

Twenty-one reports (13.0%) were published in general journals. Thirty reports (18.5%) concerned trials with more than 2 groups (8 of them having a placebo group), and the median number of randomized patients was 333 (interquartile range, 201, 656) (Table 1). A total of 139 reports (85.8%) compared different treatments (102 [87.9%] of the noninferiority and 37 [80.4%] of the equivalence reports). Twenty-three reports (14.2%) described comparisons of the same pharmacological treatment. The term noninferiority or equivalence trial was present in the title of 21 reports (13.0%) and in the abstract of 161 (99.4%).

The noninferiority or equivalence margin was specified in 156 reports (96.3%) (Table 2). This margin was relative (eg, relative risk) in 22 reports (13.6%). Justification for the margin chosen was given in 33 reports (20.4%) and was based on clinical considerations or results of a previous study in 18 reports (11.1%), statistical considerations in 5 (3.1%), and both in 10 (6.2%).

Of 127 reports (78.4%) including a sample size calculation, only 116 (91.3%) described a calculation taking into account the noninferiority or equivalence margin (Table 2). Of the 127 reports, 40 were missing some elements necessary to reproduce the calculation. In 49 reports (30.2%; 39 [33.6%] noninferiority and 10 [21.7%] equivalence), the sample size was reported to be increased because of the foreseen proportion of dropouts.

We could not determine the population analyzed in 14 reports (8.6%). A per-protocol analysis was described in 109 reports (67.3%) and ITT or modified ITT in 117 (72.2%) (Table 2). Sixty-nine articles (42.6%) reported that both per-protocol and ITT or modified ITT analyses were performed (51 [44.0%] noninferiority and 18 [39.1%] equivalence). The results of both analyses were detailed in 39 (56.5%) of those 69 reports, whereas other reports indicated only whether results were the same.

A total of 136 reports (84.0%) displayed results with a CI (95 [81.9%] noninferiority and 41 [89.1%] equivalence) (Table 2). The CI was mostly 2-sided, even for noninferiority trials (82 [70.7%] of the reports). For the 13 noninferiority trials reporting a 1-sided CI, only 3 used a 2.5% type I error rate. Of the 136 reports (84.0%) describing analysis by CI, 101 (74.3%) indicated the type I error used in the sample size calculation. Thirty-four (21.0%) described a CI whose size was not in accordance with the type I error rate used in the sample calculation (27 [23.3%] noninferiority).

In 78 reports (48.1%), results were displayed using P values, but only 39 (24.1%) of the statistical tests performed took into account the noninferiority or equivalence margin, meaning that half of these tests were for superiority purposes. In the end, 149 articles (92.0%) described results with either CIs or a statistical test taking into account the noninferiority or equivalence margin.

A subsample of 33 reports (20.4%; 24 [20.7%] noninferiority and 9 [19.6%] equivalence) satisfied the 4 requirements specific to noninferiority and equivalence trials we described previously. Moreover, if we also include justification of the margin as an equivalent requirement, this figure decreases to 7 reports (4.3%).

Of the 33 reports satisfying our quality requirements, conclusions of 4 (12.1%) were highly misleading with their claim of equivalence or noninferiority, although their results were inconclusive.

We assessed the methodologic quality of reports of randomized controlled trials of noninferiority and equivalence. Our results highlight important deficiencies in the reporting of these trials, which could be improved on in the planning stages. Table 3 points out what should be planned for and reported from noninferiority and equivalence trials. These recommendations focus on the methodologic elements specific to noninferiority and equivalence trials (ie, the definition of the noninferiority or equivalence margin, the sets of patients to analyze, and the statistical methods to use), provided the usual requirements for all randomized trials are fulfilled (eg, randomization, concealment, blinding).

Prespecifying the noninferiority or equivalence margin is necessary and has to be clinically justified.¹² This prespecification is all the more important because some reported margins were so large that they were clearly unconvincing (eg, in one report, the 95% CI of an odds ratio was required to be included in the interval from −0.33 to +3.0) and because an unjustified a priori margin will allow external reviewers to a posteriori consider it clinically questionable.

For noninferiority or equivalence trials, ITT analysis may lead to biased conclusions because of protocol violators and withdrawals. Also, dropouts and nonadherent participants from the 2 groups are potentially different, which may also bias a per-protocol analysis. Thus, both analyses are required and considered to have equal importance in drawing a conclusion.^5,14 Reporting the results of only 1 of the analyses may reflect either ignorance about noninferiority and equivalence trials or a deliberate intention to mask some of the results, potentially modifying the interpretation of the results and preventing readers from drawing definitive conclusions.

The adequacy of a noninferiority or equivalence claim could be judged only in the subsample of 33 well-reported trials in our study. However, even in this selective subsample, we identified 4 (12.1%) reports with misleading conclusions, which misidentified noninferiority or equivalence. These reports are in addition to those confusing noninferiority and equivalence, which is acknowledged to be of lesser importance. In addition, the wording used in conclusions often lacks precision in describing noninferiority and equivalence. It is surely preferable to use standard vocabulary such as treatment A “is not inferior to” (or “is equivalent to”) treatment B “with regard to the margin prespecified at Δ” rather than stating a treatment is “not substantially lower than” or “similar to” a control treatment.

Our study of noninferiority and equivalence trial reports has some limitations. First, the sample of studied reports may not be exhaustive: some trials planned with a noninferiority or equivalence aim may have been missed even though we used standard key words. Second, because we studied reports, we were unable to detect trials planned as superiority trials and secondarily reported as noninferiority or equivalence trials after failure to demonstrate superiority. We suspect that some of the articles correspond to this scenario. Discrepancies may exist between the reports and the actual protocol used,^17- 19 and some deficiencies may be due only to poor reporting (eg, the absence of elements needed for sample size calculation).

In conclusion, our findings highlight the need for improved planning, analysis, and reporting of randomized controlled trials of noninferiority and equivalence. To improve the planning and reporting of such trials, we present practical recommendations to help researchers enhance the methodologic quality and the reporting of noninferiority and equivalence trials (Table 3). We moreover stress the importance of drawing a conclusion by comparing the obtained results with the prespecified margin, which is mainly a guideline, and to use a standard vocabulary, thus avoiding potentially misleading conclusions.