Analysis and Reporting of Factorial Trials:  A Systematic Review FREE

Factorial randomized trials permit investigators to evaluate 2 (or more) interventions in a single experiment. In its simplest 2 × 2 form, shown in Table 1, participants are randomly assigned to 1 of 4 groups: 1 group receives both treatments A and B (cell AB), 1 receives only treatment A (cell A0), 1 receives only treatment B (cell B0), and the remaining group receives neither treatment A nor B (cell 00). To preserve blinding, the latter 3 may be given the corresponding placebos.

The most powerful analysis of a factorial trial is performed "at the margins" of the table: ie, the efficacy of A is determined by comparing outcomes among all patients treated with A (cells AB and A0) with those of all patients not treated with A (cells B0 and 00), and the efficacy of B by comparing cells AB and B0 with cells A0 and 00. By performing analyses "at the margins," investigators can obtain evidence about efficacy from far fewer patients than would be needed if A and B were individually tested in 2 separate trials.

Factorial trials are ideal when the 2 treatments act independently (A is equally effective whether or not the patient is receiving B, and vice-versa). In this case, they will show additive effects when used together, and "at the margins" analyses are appropriate. However, when their effects are less than additive (A is less effective when the patient is receiving B, or vice-versa), or more than additive (A is more effective when the patient is receiving B, or vice-versa), an interaction is occurring and it is not appropriate to carry out the analysis "at the margins."¹ In the presence of interactions, analyses must be carried out "inside the table": ie, the efficacy of A is determined by comparing outcomes in patients who receive only A (cell A0) with those in patients who receive neither drug (cell 00), and the efficacy of B by comparing cell B0 with cell 00.

The extent to which "at the margins" and "inside the table" analyses generate different estimates of efficacy depends on the degree of interaction between the 2 treatments. When the interaction is less than additive (ie, antagonistic), the "at the margins" analysis underestimates the efficacy of the drugs, and when the interaction is more than additive (ie, synergistic), the "at the margins" analysis overestimates efficacy. On the other hand, "inside the table" analyses use only half as many patients as "at the margins" analyses, exclude the patients in cell AB, and use patients in cell 00 twice. As a result, the confidence intervals around treatment estimates are much wider for "inside the table" than for "at the margins" analyses, and the use of the same control group (cell 00) for both analyses creates a problem with multiple comparisons.

The need for full and accurate reporting of the conduct and analysis of randomized trials has received increasing attention in recent years, accompanied by some evidence that the quality of 2-group parallel randomized trial reports is improving.^2- 3 Although factorial trials appear to have become more common (from less than 1% of the 5880 clinical trials listed on MEDLINE between 1970 and 1980 to 4.6% of the 11 479 trials listed for the last decade [F.A.M., electronic search, January 8, 2003]), their reports have received considerably less attention than have those of parallel-group randomized trials.^4- 5 This may be due to the fact that there is no MEDLINE medical subject heading (MeSH) term for factorial trials and it is therefore difficult to readily identify these trials.

Thus, we conducted a systematic review of factorial trials to examine the reporting of their rationale, methods, and analysis.

There is no MeSH term for factorial trials, and thus we searched MEDLINE, EMBASE, and the Cochrane Controlled Trials Register using the terms factorial, interaction, 2 × 2, 2 by 2, and incremental to identify factorial trials published between January 2000 and July 2002. We compared the search results with those from a hand search of all randomized trials published in December 2000 (D.G.A., written communication, January 15, 2003). This comparison revealed that our electronic search strategy did not capture some factorial trials. Thus, we conducted a detailed hand search of all trials in a defined topic area (all trials in MEDLINE 1966-2002, EMBASE 1980-2002, and the Cochrane Controlled Trials Register that used the MeSH term myocardial ischemia (exp) and were published in English). Additional relevant trials were sought by searching their references.

Two of the authors (F.A.M., S.E.S.) independently reviewed the search results and selected all publications that appeared to be factorial trials and reported treatment effects on clinically important binary outcomes (Table 2a and Table 3). Trials reporting only continuous surrogate end points (such as cholesterol or blood pressure levels, quality-of-life scores, etc) were excluded given concerns over the use of unvalidated surrogate outcomes and the inability to generate relative risk reductions for continuous outcomes.

Based on a review of the literature and a priori discussion, a list of items (including aspects of study methodology) that could impact the validity of a factorial trial's results was generated (Table 4). These items were then abstracted from each report independently (F.A.M., S.E.S.) with any discordant results resolved by consensus.

Data were extracted on the clinically important binary outcomes listed in Table 2 and Table 3. Outcomes were analyzed using the intention-to-treat principle, using the definitions and data provided in the original study publications.

Where possible, we extracted results for the 4 treatment cells in each trial (Table 1) and reanalyzed the data to determine relative risk reductions for A and B "inside the table" and "at the margins." We illustrate our analyses in Table 5 with data from the FRISC II (Fragmin and Fast Revascularization During Instability in Coronary Artery Disease II) factorial trial that compared early invasive vs noninvasive treatment (treatment A) and dalteparin vs placebo (treatment B) in 2457 patients with acute coronary syndromes.²⁰ The analyses of these data for efficacy (both "at the margins" and "inside the table") and interaction are shown in Table 6.

We examined interaction between A and B by generating an "interaction ratio" that compares the effects of each treatment in the presence and absence of the other treatment. The resulting interaction ratio and its 95% confidence interval are equivalent to an examination for heterogeneity in a meta-analysis.^54- 55 It should be noted that there is only 1 interaction ratio for any 2 × 2 factorial trial—the interaction ratio generated by comparing the effect of A in the presence and absence of B is identical to that comparing the effect of B in the presence and absence of A. As calculated here, interaction ratios above 1.0 indicate synergy and interaction ratios below 1.0 indicate antagonism. For example, in the FRISC II Trial, there is a nonsignificant trend toward synergy between the treatments (Table 6). Prior to analyzing the data in this study, we anticipated that examinations for interaction were likely to be statistically underpowered (since they are carried out "inside the table") and defined interaction ratios of 1.25 or greater and 0.8 or less as "clinically significant."^1,54

Myocardial Ischemia Factorial Trials. Of the 2283 myocardial ischemia search citations, 101 were retrieved for detailed evaluation because they appeared to be factorial from their titles or abstracts. Sixty-eight of the retrieved publications were excluded: 51 reported continuous surrogate outcomes only, 7 reported protocols or baseline demographics for ongoing trials, 6 did not fit our definition of factorial (for example, they tested 4 different interventions in a parallel design), 3 reported only 1 dimension of the factorial trial, and 1 tested a multifaceted educational intervention. The final set of 33 eligible publications described 29 unique trials (the results for A and B were reported in separate articles for 3 trials,^{23- 24,27- 28,36- 37} and 1 trial reported the efficacy outcomes in one publication and data on adverse effects in another^29- 30) (Table 2).

Non–Myocardial Ischemia Factorial Trials. Of the 174 citations identified in our electronic search for factorial trials, 92 were retrieved for detailed evaluation. Seventy-five of the retrieved publications were excluded: 48 reported continuous surrogate outcomes only, 11 reported protocols or baseline demographics for ongoing trials, 10 tested interventions to change provider or patient behavior and didn't examine patient outcomes, 4 did not fit our definition of factorial, and 2 were subgroup analyses of factorial trials already included. The final set of 17 factorial trials tested interventions for a wide variety of conditions, with the most common topic being myocardial ischemia (4 trials, all of which had been identified in our myocardial ischemia search).

Of the 524 randomized trials published in December 2000, 5 were identified as factorial (only 1 had been detected by our electronic search for factorial trials). Two of these 5 trials were excluded because they described continuous surrogate outcomes. Thus, a total of 15 factorial trials^39- 53 on topics other than myocardial ischemia were identified (Table 3).

Disagreement among the 2 reviewers regarding study eligibility occurred on 6 occasions (κ = 0.89). All disagreements were resolved by discussion.

Sensitivity of Electronic Search for Factorial Trials. Only 22 of the 29 myocardial ischemia trials found in our hand search were identified by the electronic search outlined above (76% sensitivity of electronic searching for identifying factorial trials in myocardial ischemia).

Reporting of Results.Table 4 outlines the reporting of results for these trials. The majority (36/44 [82%]) selected the factorial design for reasons of efficiency, with only 8 reports (18%) stating that their primary purpose was to examine whether combining the tested interventions resulted in incremental benefits.

Most of the trials (39/44 [89%]) reported the results of the "at the margins" analyses in their abstracts, and all but 1 reported these results in their text. Seventeen of the 44 trials (39%) reported "inside the table" analyses in their "Results" sections, and a further 12 (27%) presented the data for each cell of the 2 × 2 table such that "inside the table" analyses could be performed.

Twenty-nine trial reports (66%) specified that interactions were sought, and 26 (59%) reported using statistical methods to do so (11 used multivariate analyses, 3 used analysis of variance, 2 used the Breslow-Day test for homogeneity, 2 used χ² tests, and 8 did not specify the test used). All but 1 of these analyses were reported as P values rather than interaction ratios or odds ratios accompanied by their confidence intervals. Of these 29 trials, 22 reported the results of interaction tests and 28 stated there was no interaction between the 2 treatments (4 in the abstract and 24 in the body of the manuscript); the 29th trial prespecified an examination for interaction in the "Methods" section but did not report the results or their interpretation. Overall, 11 trials (8 in myocardial ischemia, 3 others) reported only "at the margins" analyses for efficacy, were silent about examining for interactions, and did not provide data for each cell of the 2 × 2 table that would have permitted the reader to perform "inside the table" analyses for efficacy.

Twelve trials presented data permitting an "inside the table" analysis for adverse effects. None of the 44 trials reported examining for potential interactions in adverse effects.

Quantitative Data Synthesis. After excluding the efficacy data for 2 trials with fewer than 50 patients per group and low event rates, the 27 trials that provided sufficient efficacy data for each cell of the 2 × 2 table provided a total of 31 comparisons for analysis (since 1 trial used a 3 × 2 design¹¹ and 2 others a 2 × 2 × 2 factorial design^13,39): 22 (71%) of these comparisons had interaction ratios close to 1.0, signifying little interaction between treatments A and B for efficacy (eTable 1 and eTable 2). In 5 trials ^{9- 10,21,39,42} (16% of comparisons) the treatments appeared antagonistic, with interaction ratios less than or equal to 0.8, and in 1 case⁹ the interaction was statistically significant such that the "at the margins" analyses underestimated the benefits of therapy sufficiently that treatment B appeared to be ineffective when given alone. In 2 trials (6% of comparisons),^6,41 the treatments demonstrated synergy (interaction ratios, 1.26 and 1.39, respectively), but in only 1 case was the interaction ratio statistically significant⁴¹; however, although the "at the margins" analysis overestimated the benefits of each treatment, the confidence intervals still crossed unity such that the qualitative conclusion that neither therapy was efficacious was unchanged. There was no association between the magnitude of any observed interactions and the size of the trials.

Two of the 7 interactions between treatments for adverse effects exceeded our a priori boundaries. As anticipated, in both cases the "at the margins" analysis for adverse effects overestimated the harm from either agent when both treatments had similar adverse-effect profiles (such that there were more adverse effects with A in the presence of B than without B).^7,46 Conversely, it is likely that "at the margins" analyses for adverse effects will underestimate harm from either agent in those situations where their adverse effect profiles differ.

Factorial trials are an efficient way to evaluate 2 (or more) interventions. For all but 1 of the trials in our review, the results were presented using the "at the margins" analysis appropriate for their design. However, their accurate interpretation also requires cell-by-cell results that permit efficacy and safety analyses "inside the table" as well as "at the margins"; such data were not published for one quarter of these trials. In addition, it is desirable for reports of factorial trials to include estimates of the interaction between the treatments.¹ A third of the trial reports in our sample did not do so. The variability we have demonstrated in the reporting of factorial trials supports the plans of the CONSORT (Consolidated Standards of Reporting Trials) group to widen its focus to consider the reporting of factorial trials.⁵⁶

Concerns have been expressed about the potential for unrecognized interactions between treatments in factorial trials to distort their published results and interpretations.¹ We found only 7 trials with clinically significant interaction ratios (≥1.25 or ≤0.8), only 2 trials (6% of comparisons) where this interaction was statistically significant, and only 1 trial (3%) where reliance on the "at the margins" analyses could have led to erroneous conclusions. However, even when interactions are neither statistically nor clinically significant, they still may distort the evaluation of treatment effects.⁵⁷ This is all the more important in an era of multiple efficacious therapies for many diseases (such as acute myocardial ischemia) in which questions frequently arise as to whether the benefits of these agents are additive, synergistic, or antagonistic. We believe the factorial trial remains the most valid means by which to evaluate whether combining 2 or more therapies achieves incremental benefits.

We chose to examine the quality of published reports of factorial trials in this study despite the known effects of space restrictions and editing on the completeness of such reports. Certain information is needed to judge the validity of published trials, and failure to provide this information may be associated with biased estimates of treatment effects.^58- 60 Although failure to specify an item or analysis in a trial publication does not necessarily mean that the item or analysis was not done, preliminary data suggest that there may be little difference between the conduct of studies and their published methods.⁶¹

Given the difficulties in identifying factorial trials in the absence of a MeSH term or consistent textword in electronic databases, it would be necessary to hand search the entire Cochrane Controlled Trials Register to find all factorial trials. As a compromise, we hand searched all trials in a defined clinical situation (myocardial ischemia) to identify those that were factorial and, to evaluate the generalizability of these data to other fields, examined all factorial trials that we could find by electronic searching and/or hand searching of journals published in a defined time period. We believe our findings are likely generalizable given the lack of appreciable differences between the myocardial ischemia factorial trials and those in the non–myocardial ischemia literature.

In conclusion, our study supports the assertion that factorial trials are an efficient means of concurrently evaluating 2 (or more) interventions. We have compared the 2 primary ways in which factorial trials may be analyzed ("inside the table" vs "at the margins"), proposed a method for examining for interactions between treatments in a factorial trial by generating an interaction ratio, and have illustrated how this method can be used to estimate whether the "at the margins" analyses may have overestimated or underestimated the efficacy (and adverse effects) of each agent. Finally, despite concerns about unrecognized interactions distorting the results of factorial trials, we found only 2 cases (6%) where there was a statistically significant interaction and in only 1 of these cases did this lead to a qualitative difference in interpretation. This is within the frequency expected by chance and provides some reassurance that investigators are appropriately choosing to test only those interventions that do not have potential for substantive interaction in factorial trials.