Publication Bias and Research on Passive Smoking: Title and subTitle BreakComparison of Published and Unpublished Studies FREE

A GREATER proportion of studies with statistically significant results are published than those with nonsignificant results.^{1 - 2} Publication bias is the tendency for investigators to submit manuscripts and for editors to accept them based on the statistical significance (in either direction) of the study results.^{3 - 5} Publication bias can limit access to important data and skew the results of reviews.

Some have argued that reviews concluding that passive smoking is harmful^{6 - 9} are invalid because publication bias exists.^{10 - 16} Previously, we found that 20% of peer-reviewed articles examining the health effects of passive smoking had statistically nonsignificant results.¹⁷ While our analysis of the published literature on passive smoking suggests that statistically nonsignificant results are published, it tells us little about the statistical significance of unpublished results. Therefore, in this study, we interviewed the principal investigators of passive smoking studies to determine the extent of unpublished results in this area.

Even when clinical studies with statistically nonsignificant findings are published, there is a time lag in their publication compared with studies with statistically significant findings.^{18 - 20} Thus, we hypothesized that passive smoking studies with statistically nonsignificant results would take longer to be published than studies with statistically significant results. We also hypothesized that, as for clinical trials, study design, sample size, or funding source might influence the time to publication.

Included studies were as follows: (1) in vivo studies of passive smoke exposure in humans or animals; (2) studies of the effects of passive smoking on disease, disability, or comfort; (3) original research studies; and (4) studies funded between 1981 (the year the first study on the health effects of passive smoking was published²¹ ) and 1995 for which data collection had begun by the time of the interview. Studies that analyzed passive smoking as a covariant, reviews, policy analyses, or biochemical mechanism studies were excluded.

To identify studies, we contacted 89 organizations that potentially provided extramural or intramural funding for passive smoking research. We identified these funding agencies by examining the acknowledgment sections of published papers on passive smoking,^{22 - 23} contacting experts in tobacco research, and conducting a search of a database of foundations maintained by the Foundation Center, New York, NY.

We received information from 76 (85%) of 89 organizations contacted. The response rates were 100% (12/12) for governmental agencies, 88% (60/68) for privately funded organizations, and 56% (5/9) for tobacco industry–affiliated organizations.

This study (CHR H2758-08585-05) was approved by the Committee on Human Research at the University of California, San Francisco. Principal investigators of eligible studies were invited by letter to participate in a semistructured telephone interview. During a preinterview, investigators who agreed to participate answered a short series of questions to confirm that their study met the inclusion criteria. The investigator was also asked whether he/she had been the principal investigator for any other studies of the health effects of passive smoking. These additional studies were included in the interview if they met our inclusion criteria. An appointment for the full telephone interview was scheduled.

Shortly before the scheduled interview, investigators were faxed a confirmation letter that included a list of materials to have on hand during the interview (a list of publications, the grant proposal, and any progress reports, unpublished data summaries, talks, or abstracts). All interviews were conducted by one of the authors (A.L.M.) between January 1996 and April 1997.

To determine the extent of unpublished results, we asked investigators to identify all of their published and unpublished results on the health effects of passive smoking. We also asked investigators about their publications, study design characteristics, and reason(s) for unpublished results based on an instrument used by Dickersin et al.^{1 ,24}

To test our hypothesis that the statistical significance of results, study design characteristics, and funding sources are associated with time to publication, we classified studies as "published" or "unpublished" and collected information on the variables listed below during the interviews. Interview data were supplemented with data from the granting agency or the first publication from the study.

Studies were excluded from the analysis of time to publication if (1) data analysis had not yet begun as the investigators were not able to answer our questions about the statistical significance of the data or final sample size; (2) the publications associated with the study were published before the start of the grant, as these publications could not be attributed to the funded project; or (3) the publications associated with the study contained no statistical analyses, as statistical significance could not be categorized. The remaining studies were classified as "published" if any of the results had been published in a peer-reviewed, non–peer-reviewed, or in-press publication. Studies were classified as "unpublished" if none of the results had been published or if results had been published only as abstracts, since investigators were unable to provide copies of their abstracts or the abstracts did not contain sufficient information.

Time to Publication.— Time to publication was calculated by subtracting the year funding began from the year of publication of the first original research article on the health effects of passive smoking resulting from the grant. Unpublished studies were censored at the year the interview was conducted. The funding start date was used to measure time to publication since the completion date was difficult to define.

Statistical Significance of Data.— The statistical significance of each study was classified as "statistically significant" (results indicate a statistically significant association between passive smoking and a harmful health effect); "statistically nonsignificant" (results do not indicate a statistically significant association); or "mixed" (multiple primary outcomes were measured and at least 1 was statistically significant). "Statistically significant" was defined as a P ≤.05, an odds ratio not overlapping 1, or other statistics typically characterized as significant. For unpublished studies, the categorization of statistical significance was based on the investigator's classification of his/her unpublished results. For published studies, the categorization was based on the significance of the results presented in the first original research publication as determined by 1 of the authors (A.L.M.) and a doctorate student in epidemiology.

Study Subjects.— Classified as "animals" or "humans."

Study Design.— A study was classified as "experimental" if a test regimen initiated by the investigator was administered to a treatment group, but not a control group, or as "observational" if the study was a cohort, case-control, or descriptive study.

Study Size.— Classified as "500 or less" or "more than 500" based on the distribution of sample sizes.

Health Outcome.— Classified as "respiratory effects," "lung cancer," "cardiovascular effects," "pregnancy outcome and/or birth defects," and "other" (includes studies with mixed outcomes).

Funding Source and Length.—The primary funding source was identified by the investigator and was categorized as "government," "private," "tobacco industry," or "other" and as "external" (investigator not affiliated with funding agency) or "internal." Length of funding was classified as "5 years or less" or "more than 5 years" based on the distribution of funded studies.

The χ² statistic was used to compare the characteristics of participants and refusals. Predictors of time to publication were analyzed by proportional hazards models.²⁵ When there was evidence against the proportional hazards assumptions,²⁶ P values were obtained by the nonparametric Wilcoxon test.²⁷ Multivariate models were built using stepwise addition of variables until no further additions provided a statistically significant improvement in prediction at P =.05.

We identified 105 investigators eligible for inclusion. Four investigators could not be located, 21 were found to be ineligible after completing the preinterview, and 2 were eliminated after being interviewed because their studies had no results on the health effects of passive smoking. The response rate among the remaining 78 investigators was 83% (65/78). Characteristics of participants and refusals are shown in Table 1. Of the 13 investigators who chose not to participate, 5 cited a lack of time, 3 felt uncomfortable participating, and 5 did not state a reason.

The 65 participants provided information on 84 studies. Sixty-six (79%) of these studies were identified based on the information from granting organizations and 18 (21%) were identified by the investigators.

Fifty-nine (70%) of the 84 studies had some unpublished results. The reasons stated most frequently for unpublished results were ongoing data collection or analysis (n=33 times), lack of time (n=26), and competing priorities (n=11), such as other recently funded studies, studies where passive smoking was a minor component, and career changes. Statistically nonsignificant results were cited as a reason for failure to publish for only 2 studies. One investigator stated that they chose a less prestigious journal to publish their statistically nonsignificant results. Four manuscripts (2 with statistically significant and 2 with statistically nonsignificant results) resulting from 3 studies were unpublished because they had been rejected from a journal. These manuscripts were being resubmitted.

Of the 84 identified studies, 23 were excluded from the analysis of time to publication because data analysis had not begun (n=19), the publications were published before the start of the grant (n=2), or the publications had no statistical analyses (n=2). Of the remaining 61 studies, 47 were classified as "published" and 14 as "unpublished."

Figure 1 shows that the survival curve plots for time to publication for all studies differed by the statistical significance of the study results (P=.004). The median time to publication was 5 years (95% confidence interval [CI], 4-7 years) for statistically nonsignificant studies (n=21) and 3 years (95% CI, 3-5 years) for statistically significant studies (n=33). Only 1 of the 7 studies with mixed results was published; the time to publication was 6 years.

As shown in Table 2, univariate analysis of all studies showed that statistically significant results, experimental study design, study size of 500 or less, and animals as subjects were predictive of time to publication. When the studies with human participants were analyzed separately (Table 2), only the statistical significance of the results was predictive of time to publication. Multivariate analysis of all studies indicated that statistical significance (P=.001) and study design (P=.01) were the only independent predictors of time to publication (P>.16 for all other variables). Multivariate analysis of the human studies showed that only statistical significance was predictive of time to publication (P=.007) (P>.15 for all other variables).

Our results indicate that statistically significant studies are published sooner than statistically nonsignificant studies, as reported for clinical studies.^{19 ,28} This finding cannot be attributed to differences in the sample sizes, funding sources, or health outcomes measured. Our finding of a publication delay can be partially explained by the reasons that investigators cited for not publishing.

The delay in the publication of statistically nonsignificant results has implications for conducting reviews and meta-analyses. The time lag in publication argues for regular updating of reviews, as practiced by the Cochrane Collaboration,²⁹ as statistical significance of a review could change over time. Furthermore, defining eligibility for inclusion in a review by the start date of a study could reduce bias in reviews by including unpublished results. Since 47 (77%) of the 61 studies we identified had at least some published results, researchers conducting reviews on passive smoking should be aware of these studies and seek data from them.

One limitation of our study is that we interviewed only English-speaking investigators funded by US organizations. Therefore, we may have underestimated the number of studies of the health effects of passive smoking. A second limitation is that we had to rely on principal investigators to tell us the statistical significance of their unpublished results. Finally, our ability to identify studies was limited by the inadequate record keeping and willingness to share information on the part of funding agencies or investigators. For example, only half of the tobacco-affiliated organizations responded to our multiple requests for information. Our difficulty with identifying unpublished studies supports the need to register prospectively observational studies, as has been suggested for clinical trials.³⁰