Problems in the Design and Reporting of Trials of Antifungal Agents Encountered During Meta-analysis

Meta-analyses may become biased if the data in the individual trials are reported in a biased fashion and if authors and companies are not explicit about multiple publications of the same data. Problems with unclear reporting of trial data were recently described for the antipsychotic drug risperidone¹ (manufactured by Janssen Pharmaceutica NV, Beerse, Belgium) and have been noted earlier, including instances of reports of the same trial with no common authors.^{2 - 3}

We have reported previously a meta-analysis of placebo-controlled trials of antifungal agents in patients with cancer complicated by neutropenia.^{4 - 5} We report herein briefly a meta-analysis of trials that compared the 2 most effective drugs, fluconazole and amphotericin B. The main focus of this article is not the meta-analysis, which will be described in detail elsewhere,⁶ but the unanticipated problems we encountered in its conduct.

Fluconazole is a new antifungal agent (manufactured by Pfizer Inc, New York, NY). Fluconazole is well absorbed after oral intake and is used both orally and intravenously. In contrast, amphotericin B (manufactured by Bristol-Myers Squibb, New York, NY) is poorly absorbed and is not recommended for oral use.⁷ Although most fungal infections enter the body through the gastrointestinal tract and oral amphotericin B eliminates fungi from the gastrointestinal tract in the majority of patients,^{8 - 9} there is little documentation of its clinical effect when given orally. Chemically, amphotericin B is a polyene and thus belongs to the same class of drugs as nystatin. Nystatin (manufactured by Bristol-Myers Squibb, New York, NY) is almost insoluble, is considered to be a poor drug for treatment of patients with immunodeficiency, and is not recommended for use in cancer patients with neutropenia.⁷ Hence, we were surprised to find that most trials that compared fluconazole with amphotericin B had used oral, and not intravenous, amphotericin B and that some of the results for this drug had been combined with the results for nystatin.

The primary aim of the meta-analysis was to compare the effect of fluconazole and amphotericin B on total mortality in patients with cancer complicated by neutropenia. Secondary outcomes included invasive fungal infection, colonization, need for additional (rescue) antifungal treatment, and adverse effects leading to discontinuation of therapy.

All randomized trials, irrespective of language, that compared fluconazole with amphotericin B in cancer patients with neutropenia were eligible. Letters, abstracts, and unpublished trials were also reviewed to reduce the influence of publication bias. We excluded studies that concerned solely prevention or treatment of oral candidiasis.

As described previously,^{4 - 5} we performed a comprehensive MEDLINE search from 1966 on, most recently updated in March 1998. Information about trials not registered in MEDLINE, including those unpublished, were located by contacting the manufacturers and the authors and by scanning reference lists of articles and reviews. We also scanned selected conference proceedings and searched the Cochrane Library.^{5 - 6}

Three of the reports we retrieved described large, 3-armed trials that had compared fluconazole, amphotericin B, and nystatin but had combined the outcomes for amphotericin B and nystatin in a "polyene" group. Since these trials comprised 43% of the total patients in our meta-analysis, we did not wish to exclude them. Instead, we estimated the effect of amphotericin B in the polyene trials based on findings of an additional meta-analysis of trials comparing nystatin with placebo, fluconazole, or amphotericin B. We repeated the search strategy, adding nystatin to the list of drugs. Since we suspected there would be few trials of nystatin in cancer patients with neutropenia,⁷ we included all severe diseases predisposing to fungal infection.

Details on diagnosis, drugs administered, dosage, length of follow-up, randomization and blinding methods, number of randomized patients, number of patients excluded from analysis, deaths, invasive fungal infections, colonization, use of additional (rescue) antifungal therapy, total number of dropouts, and dropouts due to adverse effects were extracted by each of us independently. We defined invasive fungal infection as a positive blood culture, esophageal candidiasis, lung infection, or microscopically confirmed deep tissue infection.^{4 - 5} Disagreements were resolved by discussion.

We used MEDLINE and abstract books to obtain the authors' most recent addresses. All authors were asked to confirm the extracted information and to answer additional questions. We specifically asked for mortality data 3 months after study entry for all patients, including those the authors had excluded after the randomization. We also sought details on the randomization process, especially whether treatment allocation was concealed (eg, central randomization, sealed envelopes, or a code provided by a pharmacy or a company).

Since the response rate to our requests for additional information on the fluconazole trials was low, we did not contact the authors for the meta-analysis of nystatin.

The outcomes were weighted by the inverse variance of the log odds ratio. A fixed-effects analysis was performed if P>.10 for the test of heterogeneity; otherwise, a random-effects analysis was made.¹⁰ The Cochrane Collaboration's statistical software was used¹¹ ; odds ratios and 95% confidence intervals (CIs) are presented.

Our searches initially indicated that 18 relevant trials had been performed.^{8 ,12 - 28} We excluded a trial in which only 22 of the 41 patients had been randomized,²² and another that was recently published as an abstract: it described 310 patients but did not report any outcome data.²⁷ We excluded a third report²³ because its results were included in a multicenter study report.¹⁹

We mailed letters to the remaining 15 authors, requesting one of them to confirm whether his trial report²⁵ was included in a multicenter study report¹⁹ and the others to check a list of data we had extracted from their study for accuracy and to inform us whether the randomization process had been concealed, whether they possessed mortality data after 3 months, and whether they were aware of any unpublished trials. Authors of trial reports of oral amphotericin were asked why they had not used intravenous amphotericin, and authors of polyene trial reports were asked why they had not reported the results for amphotericin and nystatin separately, since these drugs were not equally effective. In addition, the authors were asked to clarify uncertainties related specifically to their studies. The responses to our requests for confirmation of the extracted data and for supplementary information are shown in Table 1. Two authors from Japan^{12 ,24} responded to our first letter. After a second round of letters, we obtained answers to some of our questions from a third author.¹⁴ We contacted 3 authors in person at meetings and 1 by telephone. This led to an answer from a fourth author.²¹

One author, Marie, whose trial we identified because it was briefly mentioned in a conference abstract by Viscoli,²⁶ wrote in response to our question as to whether the study had been published that his trial was old and that the data were with the drug manufacturer. However, the trial was actually published, which we discovered when we wrote to the author of another conference abstract,²⁸ Lapierre, who sent an extensive manuscript in English and a full publication in French from 1993²⁹ that was not indexed by MEDLINE. This report had Marie as first author and Lapierre as second author. Thus, the 2 abstracts referred to the same trial.

One author, who was a Pfizer associate, did not respond.¹⁵ When we contacted Pfizer Central Research in the United Kingdom by letter and telephone, the responding person informed us that he was unable to devote the time required to do the work, and indicated that he passed on our request to colleagues in New York.

One author answered 2 questions by memory and responded that the university at which he was previously employed had refused to give him a copy of his studies.¹³ Of the remaining 7 authors (Table 1), 3 did not respond^{16 ,18 ,25} ; another noted that she did not have access to the databases because of change of affiliation⁸ ; 1 indicated that she would send the full report when it was published²⁰ ; 1 indicated that he would contact the company where the data were on file¹⁹ ; and 1 indicated that he would try to find the data.¹⁷

In 3 large trials,^{15 ,18 - 19} the patients were randomized to 3 arms: fluconazole, amphotericin B, or nystatin. However, as noted above, the results of these trials were not reported per treatment group, but were reported as if the studies had been 2-armed, comparing fluconazole with a polyene group.

The relationships between the trials and the centers contributing to them were unclear. One of the trials was reported as a multicenter study of 536 patients with Philpott-Howard as first author.¹⁹ The trial report described that patients taking amphotericin B received 2 g/d and that the results from 50 patients at 1 of the centers had been published previously. However, these patients had received 1600 mg/d.²³ Furthermore, a third report by Finke described 40 patients who received 800 mg/d.²⁵ In that article,²⁵ Finke noted that his study was part of a multicenter trial and referenced an abstract that had Brammer, identified as a Pfizer employee, as coauthor. Brammer also coauthored the report by Philpott-Howard,¹⁹ which had not included Finke among the authors and did not reference his trial report, but it mentioned Finke as a member of the Multicentre Study Group. We therefore excluded Finke's trial, assuming it was a duplicate publication. A fourth trial report of 248 patients who received "at least 2 g daily" listed Brammer as the only author.¹⁵ This report was an interim analysis but it was not referenced by newer reports. Since its methodology is very similar to that of the other trials and several of the investigators were the same as in Philpott-Howard's report, we suspected the study by Brammer¹⁵ might represent partial overlap of contributing centers or possibly duplicate publication. Because of the uncertainty, we included the report¹⁵ in our meta-analysis (exclusion of the report would make a trivial difference).

The fifth polyene trial report seemed to describe a unique study by Ninane et al,¹⁸ but it nevertheless reported that patients at 1 of the centers had received amphotericin B and nystatin simultaneously, rather than in separate treatment arms. Furthermore, a recent publication, identified after the cutoff for our search,³⁰ described a trial of 50 patients who were randomized to receive fluconazole or nystatin. This trial was 2-armed even though the article mentioned that 18 patients "were in part included in a previously reported multicentre study." This study was the 3-armed study by Ninane et al.¹⁸

These findings raise the question of whether some single-center trials had not originally been part of a multicenter trial, but had been combined in the polyene multicenter trial reports.

The meta-analysis of the nystatin trials showed that the effect of this drug is similar to that of placebo (APPENDIX Article ). Therefore, to provide a reasonable range for the estimates of the effect of amphotericin B in the 3-armed polyene studies, we performed 2 sets of analyses. In one, we used the data as reported (unadjusted analysis) while in the other, we assumed that nystatin and placebo had the same effect (adjusted analysis). Since the combined polyene group and the fluconazole group were of very similar size in all 3 studies, we assumed that equal numbers of subjects had been randomized to receive amphotericin B and nystatin. For the adjustments, we used the relative risks for amphotericin B vs placebo that we found in our previous meta-analysis^{4 - 5} : 0.41 for invasive infection. Therefore, we would expect 41 infections in patients receiving amphotericin B for every 100 infections in patients receiving nystatin; ie, the expected fraction of infections for amphotericin B in the polyene trials is 41/141 = 0.29. Thus, for a study that reported 7 infections in the polyene group,¹⁹ the expected number of infections in the amphotericin B subgroup would be 7 × 0.29 = 2.0. Doubling this number equals 4 infections for patients receiving amphotericin B, which is what we would have expected to find if all patients receiving polyenes had received this drug.

After the exclusions explained above, 13 trials were left for the meta-analysis of fluconazole vs amphotericin B. The antifungal agent was given prophylactically in 8 trials,^{8 ,12 ,14 - 15 ,17 - 19 ,24} empirically in 4,^{13 ,20 - 21 ,26} and as treatment in 1.¹⁶ The most common diseases were acute leukemia in 9 trials,^{8 ,12 ,14 - 15 ,17 - 20 ,26} bone marrow transplantation in 2,^{21 ,24} other cancers in 1,¹³ and cancer with esophageal candidiasis in 1.¹⁶

Treatment allocation was adequately concealed in 6 trials^{12 - 14 ,21 ,24 ,26} ; none of the trials were blinded. The total number of patients was 2977. The trial drugs were given intravenously to 628 patients (21%) and orally to the remaining patients (79%). The effect estimates of the trials and the full results of the meta-analysis will be published in the Cochrane Library⁶ ; a brief summary follows below.

The mortality was similar in prophylactic and empirical studies. When oral fluconazole was compared with oral polyenes, the odds ratio was 0.64 (95% CI, 0.33-1.23) in the unadjusted analysis and 0.82 (95% CI, 0.41-1.64) in the adjusted analysis. The summary odds ratio estimate for all the trials was 0.83 (95% CI, 0.62-1.11) in the unadjusted analysis and 0.87 (95% CI, 0.65-1.17) in the adjusted analysis.

When oral fluconazole was compared with oral polyenes, the odds ratio was 0.66 (95% CI, 0.27-1.60) in the unadjusted analysis and 1.14 (95% CI, 0.41-3.16) in the adjusted analysis. The summary odds ratio estimate for all the trials was 0.87 (95% CI, 0.60-1.27) in the unadjusted analysis and 0.96 (95% CI, 0.65-1.40) in the adjusted analysis.

No adjustments were necessary for the polyene studies because they did not report data on colonization. The summary odds ratio estimate for all the trials was 0.92 (95% CI, 0.54-1.55).

We experienced unexpected difficulties in obtaining responses to our requests for additional or clarifying information about studies of antifungal agents. Although we asked the authors to answer only those questions they could and all the studies had been published between 1990 and 1996, we received additional data from only 3 of the 15 authors after 2 letters. In contrast, 9 of 13 authors of reports published in the 1990s provided additional information for our previous meta-analysis of the placebo-controlled trials^{4 - 5} (P = .02 for the difference in response rate).

Support from the manufacturer of fluconazole (Pfizer, Inc) was specifically mentioned in 9 of the trial reports and in a single-center version³⁰ of an additional trial¹⁸ (Table 1), corresponding to 92% of the patients in these studies included in the meta-analysis. No sources of support from industry or public sources were listed for the other 3 trials. Two of the authors indicated that the data were with the company and 2 indicated that they did not have access to their own data because of change of affiliation. Hence, we suggest that trial investigators should keep copies of their study data.

The majority of the control patients were given oral amphotericin B, which is poorly absorbed⁷ and poorly documented.^{4 - 5 ,7} None of the 7 trial reports using oral administration offered a rationale for this decision. These trials comprised 79% of the randomized patients, which is in contrast to the placebo-controlled trials of amphotericin B in which intravenous drug was given in 4 out of 5 studies, corresponding to 86% of the patients.^{4 - 5} This difference probably could not be explained by trends in approach or setting because 2 of the trials of intravenous amphotericin B were from the same period as those of oral administration and also concerned prophylaxis.

We find it remarkable that the rationale for combining the results for amphotericin B and nystatin in a polyene group was not explained in any of the reports of the 3-armed trials. The relation between these trials and the boundaries between them were obscure. For example, the dosage of amphotericin B was not the same in various publications of the same trial. Furthermore, data from a 2-armed trial³⁰ were used in a report of a 3-armed trial,¹⁸ which suggests that patients in the 3-armed trial may not have been truly randomized to 3 arms, although the trial was described as such.

When the trials were planned and conducted, nystatin was recognized as being an ineffective drug for invasive fungal infection.^{7 ,15} For example, Brammer, the author of 1 of the polyene reports¹⁵ that was also the first published comparison of fluconazole and amphotericin B, wrote that "there has been no convincing evidence to demonstrate that [oral preparations of nystatin and amphotericin B] reduce the incidence of systemic invasion by Candida yeasts."¹⁵

We believe we have shown that the polyene trials are biased in favor of fluconazole. First, nystatin is no better than placebo when given to patients with severe immunodeficiency, such as those with cancer complicated by neutropenia (Appendix Article ). Second, the difference between fluconazole and nystatin, odds ratio 0.27 for fungal infection and 0.19 for colonization (0.40 if the 2 treatment trials are excluded [Appendix Article ]), is similar to the difference between fluconazole and placebo, which we have reported previously (0.38 and 0.31, respectively).⁵ Another indication that oral amphotericin B and nystatin are not the same is shown by the number of dropouts due to adverse effects. More patients dropped out while taking oral amphotericin B than fluconazole, whereas the opposite was noted for oral polyenes, and the difference between the 2 odds ratios is significant (P = .002, normal approximation) (details to be published in the Cochrane Library⁶ ).

The effect of our adjustments for this bias was seen most clearly for invasive fungal infection, which was the only variable for which there were data from all three 3-armed trials. The odds ratio changed from 0.66 to 1.14 after the adjustments, which is in better agreement with the effect noted in the other studies^{8 ,12 ,17 ,24} that had compared oral drugs (odds ratio, 1.02).

However, the bias could be more pronounced than what we found. For example, in a similar meta-analysis recently presented at an international congress,³¹ azoles (mainly fluconazole) were reported to be considerably and significantly better than polyenes, with odds ratios of 0.59 (95% CI, 0.40-0.87) for reducing superficial fungal infection and 0.44 (95% CI, 0.27-0.71) for reducing definite invasive infection. This meta-analysis had a Pfizer associate among the authors, so it would have been possible to report the results for amphotericin B separately.

Another example of discrepant meta-analyses has been encountered for the selective serotonin reuptake inhibitor fluoxetine (Prozac, Eli Lilly and Company, Indianapolis, Ind). A similar dropout rate for this drug as for tricyclic antidepressants (P = .40) was reported in 1 meta-analysis,^{32 - 33} whereas a second meta-analysis, published as a correspondence by a company employee in response to the first meta-analysis, found a marked difference in favor of fluoxetine (P<.001).³⁴ However, the second meta-analysis contained fewer patients than the first, had less power, and included unpublished "data on file," which are usually less favorable for new treatments than published ones.^{35 - 36} As illustrated by this example and by our own meta-analysis, we believe readers should be skeptical about meta-analyses that contain unpublished data to which only the sponsor has access.

Such experiences may create mixed feelings for the patients and may detract from their willingness to volunteer for clinical trials. Study subjects often participate in clinical research for the benefit of future patients and they have a right to expect that the results can be trusted and openly discussed. Since individual trials are rarely conclusive, we suggest that it would be valuable for patients if investigators, institutions, and pharmaceutical companies were more helpful to those conducting meta-analyses.