When Not Being Superior May Not Be Good Enough

In this issue of JAMA, van Ruler and colleagues¹ report the results of a multicenter randomized trial comparing 2 strategies of operative management for patients with severe peritonitis. Although the study did not reveal a statistically significant benefit of an on-demand relaparotomy strategy over a planned relaparotomy approach in terms of death or major morbidity, there were fewer relaparotomies, shorter hospital and intensive care unit stays, and lower costs favoring the on-demand approach. The authors suggest that the on-demand approach may be the preferred strategy for patients with severe peritonitis.

The design of this trial was informed by prior work suggesting better outcomes associated with an on-demand strategy.^{2 - 5} Accordingly, the authors hypothesized that the proportion of adverse events among the on-demand group would be 16% lower than among the planned relaparotomy group. This superiority hypothesis (that on-demand is a better strategy) can be tested using 2 methods (1- or 2-sided testing) based on statistical assumptions about how to assess variability. The decision to accept or reject a hypothesis of superiority for either the on-demand or planned laparotomy strategy is achieved using a 2-sided testing design. Alternatively, a 1-sided design will only allow investigators to accept or reject a hypothesis of superiority for the on-demand strategy.

This study used a 1-sided test yielding a nonsignificant result, leaving 3 possible scenarios to explain this finding: (1) the 8% observed difference favoring the on-demand strategy is the true difference in outcome between an on-demand and planned relaparotomy approach, but the study did not have enough participants to distinguish this 8% difference from chance alone (underpowered); (2) the efficacy of the on-demand and planned strategies are essentially equivalent, and the 8% difference observed in this study is due to random variability; or (3) the efficacy of the on-demand approach is actually inferior to the planned strategy.

Inspection of the confidence interval for the risk difference in the primary end point (−5% to 20%) shows how all 3 scenarios are possible because the on-demand approach may have a 5% higher, an equivalent, or a 20% lower risk of adverse events compared with the planned strategy. With 1-sided hypothesis testing, lack of evidence to support superiority does not establish equivalence (also known as noninferiority) because the possibility of inferiority cannot be excluded.

Several articles have discussed the rationale for, appropriate conduct of, and reporting of noninferiority trials.^{6 - 9} The development of this methodology emerged in response to challenges faced by regulatory agencies evaluating new pharmaceutical agents with proposed advantages over standard therapy, such as better safety, tolerability, convenience, and lower costs. Approval of the new agent would only be appropriate if the new drug was as efficacious as standard therapy. Demonstrating a zero difference in outcomes between competing therapies is impossible because infinitely large samples would be required. Instead, the greatest tolerable difference favoring standard over new therapy (referred to as the noninferiority margin) is designated a priori and then tested. Choosing this difference is ideally informed by the results of a meta-analysis of prior randomized trials of standard treatment vs placebo and expert clinical judgment about the maximum decrement in efficacy allowable to experience nonefficacy-related advantages of new therapy (such as cost savings).

Noninferiority trials conducted in surgical patients and settings pose several challenges. For instance, the noninferiority margin between an alternative and standard surgical strategy will likely be informed by observational studies and expert opinion—each of which are subject to various sources of bias. Because of ethical and practical constraints, placebo-controlled trials of surgical interventions are rare, and therefore calculating a reasonable noninferiority margin is challenging.

Another problem is that noninferiority trials generally require larger samples than superiority trials. For instance, if van Ruler an colleagues designed a noninferiority trial of on-demand vs planned relaparotomy, 2250 patients would have been needed rather than the 231 patients who were actually recruited (sample size based on a noninferiority margin of 5%, 1-sided α of .05, 80% power, no drop outs, and an event rate of 65% in the planned relaparotomy group). Although there are examples of successful surgical noninferiority trials,^{10 - 12} the relatively low incidence of certain surgical conditions (including severe peritonitis) makes conducting such studies impractical because of the need to accrue a sufficient number of patients in a short time frame, across a reasonable number of centers, and with increasingly limited federally sponsored support.

Data from the well-designed and conducted superiority trial of van Ruler and colleagues¹ will inform the design of future studies aiming to establish the superiority, inferiority, or noninferiority of on-demand compared with planned relaparotomy. While the negative results of this superiority trial may not be enough to rule out alternative interpretations, the results are consistent with the notion that an on-demand relaparotomy approach may improve outcomes and save health care resources.