Association of Funding and Conclusions in Randomized Drug Trials:  A Reflection of Treatment Effect or Adverse Events? FREE

Empirical evidence shows that conclusions in randomized trials are more positive toward experimental interventions if funded by for-profit organizations.^1- 7 Three studies found this association in randomized trials published in high-impact journals.^1- 3 Two studies reached similar results in randomized trials on arthritis⁴ and myeloma.⁵ Two recent systematic reviews^6- 7 highlight the external validity of these findings. It is not known whether this association reflects the quantitative trial results.⁸ More positive conclusions in trials funded by for-profit organizations could reflect either more beneficial treatment effects or less frequent occurrence of adverse events. None of the previous studies^1- 7 assessed these aspects. Furthermore, previous studies included relatively heterogeneous trial cohorts. This case mix could confound the findings. It is possible that the association simply reflects that trials funded by for-profit organizations assess the most effective interventions.

The influence of methodological quality, type of control intervention, sample size, and disease area also could be important. These variables were assessed only in 1 study but did not appear to explain the association between funding and conclusions.³ Moreover, financial interests may influence the decision to submit trials with positive results to high-impact journals.

We assessed whether an association between funding and conclusions in randomized drug trials reflects the magnitude of the treatment effect or occurrence of adverse events. Secondary objectives were to explore the impact of methodological quality, type of control intervention, size of the trial, year of publication, or publication in high-impact journals on this association.

We selected all randomized trials included in eligible meta-analyses from a random sample of Cochrane reviews obtained in May 2001 and extracted data on conclusions, funding, treatment effect, adverse events, and additional confounding factors. The primary outcome for each trial was selected according to the meta-analysis including the trial. We analyzed whether an association between funding and conclusions reflected treatment effects or adverse events.

We selected meta-analyses of randomized trials because within each meta-analysis, trials assessed comparable treatments for specific diseases. This allowed us to adjust for disease areas and type of experimental and control intervention. Meta-analyses of binary outcomes were selected to adjust our analyses for the treatment effect estimated by an odds ratio (OR). To reduce the number of empty cells in our analyses, the meta-analyses had to include at least 5 full-paper trials. To examine the impact of trial quality, the meta-analyses had to include at least 1 trial with and 1 trial without adequate allocation concealment.^9- 11 Our inclusion criteria were adopted from previous methodological studies.^9,11- 12 We included all full-paper trials that were contained in eligible meta-analyses.

Based on a pilot study of 100 systematic reviews, we estimated that about 25% of all reviews published in the Cochrane Library would contain an eligible meta-analysis. We also estimated that we had to include about 40 reviews to obtain a sample of about 500 trials. Previous studies^1- 3,5 have indicated that this number would provide an acceptable risk of type II error. The number of systematic reviews in the Cochrane Library 2001, Issue 2 was 1081.¹³ To obtain about 40 reviews, we used a computer-generated list of random numbers in blocks of 26 and randomly selected 4 trials from each block. This resulted in 167 systematic reviews, which were screened by one author (B.A.-N.) for eligible meta-analyses.

Two of the authors (B.A.-N., L.L.K.) independently selected the primary binary outcome for each included meta-analysis. The primary binary outcome from each meta-analysis was considered the primary outcome for all trials included in each meta-analysis. If it was not explicitly reported or several primary outcomes were listed, we chose the most clinically relevant outcome, assessed by the largest number of trials.

All data were extracted from the original trial reports. We defined conclusions as the interpretation of the extent to which the overall trial results favored the experimental over the control intervention. We graded conclusions according to the phrasing in the abstract and the summarized conclusion on a scale of 1 to 6 points (Box).¹⁴ The higher the score, the more positive the conclusion toward the experimental intervention. Because the score did not conform to a normal distribution, we divided conclusions into whether the experimental drug was recommended as the treatment of choice without disclaimers (6 points) or not (1-5 points).

6 Points

Experimental intervention highly preferred and should now be considered the standard intervention in all patients, or similar

5 Points

Experimental intervention preferred to control, but further trials still indicated, experimental may be more costly, or similar

4 Points

Experimental and control intervention about equal, but the experimental cheaper, easier to administer, or similar minor advantage

3 Points

Experimental and control intervention about equal, but the control may be cheaper, easier to administer, or similar minor advantage

2 Points

Control intervention preferred to experimental intervention, but experimental intervention might be promising under some circumstances, or similar

1 Point

Control intervention highly preferred and should now be considered the standard intervention in all patients, or similar

We extracted the sources of funding from the text, statements of sources of support, authors' affiliations, and acknowledgments. Funding sources were classified as nonprofit organizations, not reported, both nonprofit and for-profit organizations, or for-profit organizations. For-profit organizations were defined as companies that might acquire financial gain or loss depending on the outcome of the trial. Funding included provision of grants, study materials (drug, placebo, assay kits, or similar materials), or manpower (authorship, statistical analysis, or other assistance). For the primary binary outcome measure, we extracted the number of outcomes and participants in the experimental and control groups.

We classified the occurrence of adverse events as (1) no significant difference between experimental and control group, (2) significantly more frequent in the control group, (3) significantly more frequent in the experimental group, or (4) not reported.

We assessed methodological quality from the original trial reports and any additional information provided in the Cochrane review. We assessed the following 3 components^10- 11: generation of the allocation sequence (classified as adequate if based on a table of random numbers, computer-generated, or similar), allocation concealment (classified as adequate if based on central randomization, identical coded drug boxes, sealed envelopes, or similar), and double blinding (classified as adequate if the trial was described as double blind). We extracted the type of control intervention (placebo/no intervention or active intervention), the number of patients randomized, and whether a preset sample size was estimated and reached. We registered the meta-analysis in which the trial was included, the year of publication, and whether the trial was published in a high-impact journal (impact factor ≥6).^1- 3

Two authors (B.A.-N., W.C.) independently extracted data from each trial in an unblinded manner. Consensus was achieved before data entry. A third author (C.G.) arbitrated disagreements. A fourth author (L.L.K.), who was blinded with regard to funding, extracted conclusions in a random sample of 60 trials. The intraclass correlation coefficient between blinded and unblinded assessment of conclusions was 0.93 (95% confidence interval [CI], 0.89-0.96).

For each trial, we estimated the OR of an unfavorable outcome (eg, mortality). The SE of the logarithm of the OR was calculated as a measure of uncertainty.¹⁵ We calculated a z score (log OR/SE to log OR) as a measure of treatment effect.¹⁵ The z score combines the magnitude of the point estimate (log OR) with the level of uncertainty (SE).

We used the Kruskal-Wallis test for testing the overall null hypothesis of no association between funding and conclusions assessed on the continuous scale. We used a cutoff value between 5 and 6 points to divide trials into whether the conclusions recommended the experimental drug as the treatment of choice or not. We used logistic regression to assess the association between funding and conclusions while adjusting for treatment effect, adverse events, and other potentially confounding trial variables (methodological quality, sample size, whether preset sample size was estimated and reached, meta-analysis, year of publication, and journal impact factor). The logistic regression model was fit using conclusions as the dependent variable and including trial variables in a forward stepwise procedure. Meta-analysis was kept in the model irrespective of statistical significance to adjust for the disease area and type of drug and control intervention. All other variables were excluded if P>.05. The appropriateness of the logistic regression models was confirmed by the Hosmer-Lemeshow test.¹⁶ All P values were 2-tailed and significance was defined as P<.05. Analyses were performed in SPSS version 11.0 for Windows (SPSS Inc, Chicago, Ill).

From our random sample of 167 Cochrane reviews, we excluded 126 that included fewer than 5 full-paper randomized trials in a meta-analysis (n = 105), included only trials with adequate (n = 6) or inadequate allocation concealment (n = 13), or did not assess a binary outcome (n = 2). The remaining 41 reviews contained meta-analyses, which included 523 trials, that met our inclusion criteria. Sixteen of these reviews, which included 153 trials, assessed nonpharmacological interventions. Initial analyses revealed that only 4 (3%) of these trials were funded by for-profit organizations. We therefore limited our analyses to the 370 drug trials (references available on request from the authors) from 25 reviews.^17- 41

In most trials, conclusions favored the experimental drug (median score [interquartile range], 5 [4-6]). In 36% of trials (n = 135), the experimental drug was recommended as the treatment of choice (6 points).

Eighteen percent of trials (n = 67) were funded by nonprofit organizations and in 29% (n = 106) funding was not reported. Fourteen percent of trials (n = 51) were funded by both nonprofit and for-profit organizations and 39% (n = 146) by for-profit organizations alone. The treatment effect assessed by the mean (SD) z score was –1.39 (1.90) (range, –10.99 to 5.45). In 50% of the trials (n = 185), the occurrence of adverse events did not differ significantly between the intervention groups, and in 5% (n = 20) the occurrence was significantly higher in the control than that in the experimental group. In 16% of trials (n = 60), the occurrence of adverse events was significantly higher in the experimental group, and in 28% (n = 105) adverse events were not reported.

Adequate generation of the allocation sequence was reported in 28% of trials (n = 105), adequate allocation concealment in 22% (n = 82), and 63% (n = 234) were double blind. In 76% of trials (n = 283), the control intervention was placebo or no intervention. The median number of patients randomized was 98 (range, 10-82 892). Preset sample size was estimated and reached in 21% of trials (n = 76).

The disease areas were intensive care (n = 85), smoking cessation (n = 78), respiratory diseases (n = 54), gynecology/obstetrics (n = 48), gastroenterology (n = 33), neurology (n = 26), psychiatry (n = 13), infectious diseases (n = 12), rheumatology (n = 9), nephrology (n = 6), and dermatology (n = 6). The primary outcome measures were smoking cessation (n = 78), mortality (n = 64), blood transfusion (n = 61), withdrawals (n = 33), dysphagia (n = 23), endometritis (n = 20), parasitemia (n = 17), depression (n = 13), admission to hospital (n = 11), bronchiolitis (n = 8), neurologic deficit (n = 8), cesarean delivery (n = 7), warts (n = 6), cytomegalovirus disease (n = 6), pregnancy (n = 5), bacterial vaginosis (n = 5), and asthma (n = 5).

The year of publication ranged from 1971 to 2000 with 1990 (5.6 years) as the mean (SD) publication year. Eighteen percent of trials (n = 65) were published in high-impact journals.

The conclusions in trials stratified by funding are shown in Table 1. Conclusions were significantly more favorable toward experimental drugs in trials funded by for-profit organizations compared with those of trials funded by other sources (P<.001, Kruskal-Wallis test). The proportion of trials in which conclusions recommended the experimental drug as the treatment of choice (6 points) was significantly higher among trials with for-profit funding compared with that of other trials (P<.001, χ² test).

The distributions of the potential confounding variables stratified by funding are shown in Table 2. Funding by for-profit organizations alone or by for-profit and nonprofit organizations was associated with more complete reporting of adverse events, more adverse events in the experimental group, more frequent report of adequate allocation concealment and double blinding, and more frequent use of placebo or no treatment as control intervention. Funding by both nonprofit and for-profit organizations also was associated with a larger sample size and publication in high-impact journals. No significant difference was observed between the groups regarding the treatment effect, adequate generation of the allocation sequence, or whether a preset sample size had been estimated and reached.

The logistic regression analyses showed that funding, treatment effect, and double blinding were significantly associated with conclusions (Table 3). None of the remaining variables were associated significantly with conclusions. After adjusting for the treatment effect and double blinding, conclusions were significantly more likely to recommend the experimental drug as treatment of choice in trials funded by for-profit organizations alone compared with trials funded by nonprofit organizations (OR, 5.3; 95% CI, 2.0-14.4). Compared with trials funded by nonprofit organizations, conclusions were not significantly different in trials not reporting funding or trials funded by both nonprofit and for-profit organizations. The likelihood of recommending the experimental drug as the treatment of choice decreased with decreasing treatment effect (OR, 0.6; 95% CI, 0.5-0.7) and increased with adequate double blinding (OR, 2.9; 95% CI 1.4-6.0) (Table 3).

In randomized drug trials from a randomly selected sample of reviews published in the Cochrane Library,¹³ we found that conclusions of trials were significantly more likely to recommend the experimental drug as the treatment of choice if trials were funded by for-profit organizations. This result is in accordance with previous studies.^1- 7 The present study adds to previous evidence by showing that this association does not reflect the quantitative trial results; neither the magnitude of the treatment effect nor the occurrence of adverse events could explain the association.

We selected meta-analyses from a random sample of Cochrane reviews because the Cochrane Collaboration is a nonprofit organization that aims to minimize the influence of financial and other competing interests. Furthermore, Cochrane reviews appear to be of higher quality and are less prone to bias than reviews published in traditional medical journals.^42- 44 We excluded a high (75%) but expected number of the randomly selected reviews because they did not contain a meta-analysis that fulfilled our inclusion criteria. Furthermore, only 3% of the nonpharmacological trials in our original sample were funded by for-profit organizations. Because of the low frequency of for-profit funding, these trials would not contribute valuable information, but rather introduce noise into the analyses.⁴⁵ Therefore, these trials also were excluded. Considering that more than 360 000 randomized clinical trials have been published,⁴⁶ our results are based on a small sample. These factors may reduce the external validity of our study. However, our sample was obtained randomly from the Cochrane Library, which covers all medical areas.¹³ The included trials covered a variety of medical areas, drugs, publication years, and journals. This variability supports the external validity of our findings.

Within each meta-analysis, trials addressed the same clinical question, compared the same treatments, and provided data on the same outcome measure. We therefore were able to adjust our analyses for both disease areas and type of experimental and control treatments. Therefore, the clinical importance of the treatment corresponds to the magnitude of the treatment effect. Together the point estimate (OR) and the level of uncertainty (SE) provide information about the treatment effect. We used a z score to capture these 2 aspects in 1 variable. We found similar results when including the OR and the SE of the treatment effect separately in the logistic regression model (data not shown). The magnitude of the treatment effect did not explain the association between funding and conclusions.

To estimate the treatment effect, we selected the primary binary outcome specified in the meta-analysis including the trial. We did this to obtain clinically important and homogeneous data. Many randomized trial reports do not specify the primary outcome measures,⁴⁷ use surrogate outcomes,^48- 51 or use multiple primary outcomes.⁵² Cochrane reviews are based on prespecified, peer-reviewed, and published protocols in which the outcome measures most clinically relevant to patients are selected.⁵³ Accordingly, the selection of the primary outcome measure in Cochrane reviews aims to be unbiased and not data driven.⁵²

The favoring of experimental drugs in trials funded by for-profit organizations did not appear to reflect the occurrence of adverse events. Compared with trials funded by nonprofit organizations, trials funded by for-profit organizations reported a significantly higher number of adverse events in the experimental arm. This might reflect differences in the quality of reporting. Possibly, drug companies are more focused on reporting of adverse events because of use of Good Clinical Practice guidelines.⁵⁴

We assessed the effect of several potential confounders. These were selected according to previous evidence and theoretical considerations. Several of the confounders had some overlap (eg, the treatment effect and the use of inactive control, double blinding and the use of placebo as control intervention). However, this overlap only increased the SEs of the logistic regression model but did not invalidate the model. Our results indicated that the methodological quality, type of control intervention, sample size, whether a preset sample size had been estimated and reached, year of publication, or publication in high-impact journals did not explain the association between funding and conclusions.

Trials funded by for-profit organizations had better methodological quality than trials funded by nonprofit organizations regarding allocation concealment and double blinding. This finding is in accordance with previous studies.^5,7,55 We are aware that discrepancies might occur between report and conduct of trials.⁵⁶ The Cochrane Collaboration recommends that reviewers correspond with the primary investigators of trials and companies to obtain information about central methodological aspects.⁵³ We based our assessment of quality on both the trial reports and any additional information provided in the Cochrane reviews.^17- 41

We assessed the reported conclusions using a scale developed by Gilbert et al.¹⁴ The lack of validation of this scale may be criticized.²⁰ However, the scale has been used in 3 studies,^3,5,57 and we found that it had high face and content validity as well as high reliability. One study analyzed the reported conclusions on the continuous scale.³ Other studies divided the reported conclusions into "positive" or "negative."^1- 2,4- 5 In the present study, only 20% of the trials scored 1 to 3 points. This skewed distribution of data supported the use of a cutoff value between 5 and 6 points. This allowed us to explore whether conclusions were more likely to recommend the experimental drug as the treatment of choice without disclaimers. Such conclusions must have a considerable impact on clinical decision making. Sensitivity analyses revealed that selecting another cutoff (eg, between 4 and 5 points) gave similar results. This increases the robustness of the evidence and supports the existence of an association between funding and conclusions.^1- 7

A potential weakness of the present study is that the reported conclusions were assessed unblinded with regard to the source of funding. However, we found high interobserver agreement between blinded and unblinded assessment. This concurs with previous findings^3,58 and suggests that blinding would not significantly change our conclusions.

Sponsor involvement and influence on the conduct and reporting of a trial varies. The degree of influence is difficult to assess from trial reports. We combined trials funded entirely by for-profit organizations with trials in which only drugs and placebo were provided. Sensitivity analyses showed that the significant association between funding and conclusions was present both in the group of trials having only obtained for-profit funding in the form of drug and placebo and in the group of trials having obtained more substantial for-profit funding (data not shown).

A large proportion of trials did not report the sources of funding. It is likely that funding by for-profit organizations is underreported in trials. However, the degree of underreporting is not possible to assess. We found that about 50% of the included trials were funded solely or in part by for-profit organizations. This is in accordance with a recent review, in which the median proportion of trials receiving for-profit funding was 39% (interquartile range, 23%-64%).⁷

Conclusions reflect a trade-off between efficacy, safety, and cost-effectiveness. We did not assess the impact of cost-effectiveness on conclusions. Economic analyses are rarely included in randomized trials^59- 60 and new interventions are generally more expensive than conventional ones. Friedberg et al⁶¹ showed that studies in oncology funded by for-profit organizations were nearly 8 times less likely to reach unfavorable conclusions regarding economic assessments of experimental interventions than studies funded by nonprofit organizations.

We based our analyses on all full-paper trials that were included in the eligible meta-analyses. Trials that were only published as abstracts or letters were excluded as they very rarely contained information on funding and/or trial quality. We did not assess the studies that the authors of the Cochrane reviews had excluded from meta-analyses. The reasons for exclusion were described in all reviews. The majority of studies were excluded because they were not randomized trials. It is possible that these studies estimated a greater treatment effect because lack of randomization increases the risk of selection bias. However, it is uncertain whether the design of a study (randomized or nonrandomized) has a significant effect on the association between funding and conclusions. Our study was not designed to address this aspect.

The present study cannot show the causes or consequences of the association between for-profit funding and conclusions favoring the experimental drug. Our study was designed to assess if an association reflected the quantitative trial results. We found a significant association between conclusions and the estimated treatment effect. The likelihood of recommending the experimental drug as the treatment of choice increased significantly with increasing treatment effect (ie, decreasing z scores). However, this did not appear to explain the association between funding and conclusions. We found no significant difference in treatment effect between trials stratified according to funding. This is in accordance with a recent pilot study.⁶² Our findings oppose the suggestion that conclusions are more likely to be positive if funded by for-profit organizations because these trials should be more likely to reach positive results.⁸

Likely explanations for the association could be violation of the uncertainty principle, publication bias, emphasis on subgroup or secondary outcome analyses, or bias in drawing conclusions. Violation of the uncertainty principle could occur if for-profit organizations were more prone to sponsor trials that were likely to favor the experimental drug.⁵ A main objective of the pharmaceutical industry is to acquire financial gain. After having conducted exploratory randomized trials in phase 2 drug development, several confirmatory randomized phase 3 (proof of concept) trials usually are launched. Such trials may have a higher likelihood of favoring the experimental drug. Publication bias also has been suggested as a possible explanation.^1- 3,5,7,63 Concern has been raised that for-profit organizations might discontinue ongoing studies if accumulating results appear negative or if they avoid the publication of negative studies.^1,63- 64 Because of the design of the present study, we cannot refute or confirm these queries. We found no significant difference in the estimated intervention effect of the primary outcome measure between groups of trials stratified according to funding. It is possible, that the favoring of experimental interventions in conclusions of trials funded by for-profit organizations was due to emphasis of results from surrogate outcomes,⁴⁹ secondary outcome measure analyses, or subgroup analyses.^65- 66 Finally, the association between funding and conclusions might reflect a biased interpretation of the overall trial results. This potential bias could be due to financial conflicts of interest.⁶⁷

In principle, about half of all trials should favor the control intervention.⁵ We found that trial results and conclusions rarely favored the control. This finding concurs with previous studies.^{1- 4,6,57,62,64,68} The combined evidence underlines the need for an international register of all initiated randomized clinical trials. Registration would enable the public to follow the development of drugs from the beginning of phase 2 trials. Furthermore, data from unpublished randomized trials irrespective of funding source or results could be contained in an international register available to the public.^69- 70 The Consolidated Standards of Reporting Trials (CONSORT) statement⁷¹ could consider the importance of reporting of funding, as suggested by the International Committee of Medical Journal Editors.⁷² Our study suggests that editors, peer reviewers, and readers of trial reports should evaluate carefully the trial data to determine if the reported conclusions are supported by data.