0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Peer Review Congress |

Publication Bias and Research on Passive Smoking:  Comparison of Published and Unpublished Studies FREE

Anastasia L. Misakian; Lisa A. Bero, PhD
[+] Author Affiliations

From the Institute for Health Policy Studies (Ms Misakian and Dr Bero) and the Department of Clinical Pharmacy (Dr Bero), University of California, San Francisco.


JAMA. 1998;280(3):250-253. doi:10.1001/jama.280.3.250.
Text Size: A A A
Published online

Context.— The results of reviews may be biased by delays in publication and failure to publish nonsignificant results.

Objective.— To determine the extent of unpublished results on the health effects of passive smoking and whether passive smoking studies with statistically nonsignificant results would have longer time to publication than those with statistically significant results.

Design.— Semistructured telephone interviews of principal investigators of published or unpublished studies funded between 1981 and 1995, identified by information obtained from 76 (85%) of 89 organizations contacted that potentially funded research on passive smoking.

Participants.— Seventy-eight investigators were eligible and could be located; 65 (83%) responded. They had conducted 61 studies of the health effects of passive smoke in humans or animals between 1981 and 1995 that met the criteria for the analysis of time to publication.

Main Outcome Measure.— Time to publication for published studies and statistical significance of results of published and unpublished studies.

Results.— Fourteen of the 61 studies were unpublished. Median time to publication was 5 years (95% confidence interval [CI], 4-7 years) for statistically nonsignificant studies and 3 years (95% CI, 3-5 years) for statistically significant studies (P=.004). Statistically significant results (P=.004), experimental study design (P=.01), study size less than or equal to 500 (P=.01), and animals as subjects (P=.03) were predictive of time to publication. When the studies with human participants were analyzed separately, only statistically significant data were predictive of publication (P=.007). Multivariate analysis of all studies indicated that statistical significance (P=.001) and study design (P=.01) were the only independent predictors of time to publication, while for the human studies only statistical significance was predictive of publication (P=.007).

Conclusion.— There is a publication delay for passive smoking studies with nonsignificant results compared with those with significant results.

Figures in this Article

A GREATER proportion of studies with statistically significant results are published than those with nonsignificant results.1,2 Publication bias is the tendency for investigators to submit manuscripts and for editors to accept them based on the statistical significance (in either direction) of the study results.35 Publication bias can limit access to important data and skew the results of reviews.

Some have argued that reviews concluding that passive smoking is harmful69 are invalid because publication bias exists.1016 Previously, we found that 20% of peer-reviewed articles examining the health effects of passive smoking had statistically nonsignificant results.17 While our analysis of the published literature on passive smoking suggests that statistically nonsignificant results are published, it tells us little about the statistical significance of unpublished results. Therefore, in this study, we interviewed the principal investigators of passive smoking studies to determine the extent of unpublished results in this area.

Even when clinical studies with statistically nonsignificant findings are published, there is a time lag in their publication compared with studies with statistically significant findings.1820 Thus, we hypothesized that passive smoking studies with statistically nonsignificant results would take longer to be published than studies with statistically significant results. We also hypothesized that, as for clinical trials, study design, sample size, or funding source might influence the time to publication.

Identification of Passive Smoking Research Studies

Included studies were as follows: (1) in vivo studies of passive smoke exposure in humans or animals; (2) studies of the effects of passive smoking on disease, disability, or comfort; (3) original research studies; and (4) studies funded between 1981 (the year the first study on the health effects of passive smoking was published21) and 1995 for which data collection had begun by the time of the interview. Studies that analyzed passive smoking as a covariant, reviews, policy analyses, or biochemical mechanism studies were excluded.

To identify studies, we contacted 89 organizations that potentially provided extramural or intramural funding for passive smoking research. We identified these funding agencies by examining the acknowledgment sections of published papers on passive smoking,22,23 contacting experts in tobacco research, and conducting a search of a database of foundations maintained by the Foundation Center, New York, NY.

We received information from 76 (85%) of 89 organizations contacted. The response rates were 100% (12/12) for governmental agencies, 88% (60/68) for privately funded organizations, and 56% (5/9) for tobacco industry–affiliated organizations.

Interviews

This study (CHR H2758-08585-05) was approved by the Committee on Human Research at the University of California, San Francisco. Principal investigators of eligible studies were invited by letter to participate in a semistructured telephone interview. During a preinterview, investigators who agreed to participate answered a short series of questions to confirm that their study met the inclusion criteria. The investigator was also asked whether he/she had been the principal investigator for any other studies of the health effects of passive smoking. These additional studies were included in the interview if they met our inclusion criteria. An appointment for the full telephone interview was scheduled.

Shortly before the scheduled interview, investigators were faxed a confirmation letter that included a list of materials to have on hand during the interview (a list of publications, the grant proposal, and any progress reports, unpublished data summaries, talks, or abstracts). All interviews were conducted by one of the authors (A.L.M.) between January 1996 and April 1997.

To determine the extent of unpublished results, we asked investigators to identify all of their published and unpublished results on the health effects of passive smoking. We also asked investigators about their publications, study design characteristics, and reason(s) for unpublished results based on an instrument used by Dickersin et al.1,24

Analysis of Time to Publication

To test our hypothesis that the statistical significance of results, study design characteristics, and funding sources are associated with time to publication, we classified studies as "published" or "unpublished" and collected information on the variables listed below during the interviews. Interview data were supplemented with data from the granting agency or the first publication from the study.

Studies were excluded from the analysis of time to publication if (1) data analysis had not yet begun as the investigators were not able to answer our questions about the statistical significance of the data or final sample size; (2) the publications associated with the study were published before the start of the grant, as these publications could not be attributed to the funded project; or (3) the publications associated with the study contained no statistical analyses, as statistical significance could not be categorized. The remaining studies were classified as "published" if any of the results had been published in a peer-reviewed, non–peer-reviewed, or in-press publication. Studies were classified as "unpublished" if none of the results had been published or if results had been published only as abstracts, since investigators were unable to provide copies of their abstracts or the abstracts did not contain sufficient information.

Variables

Time to Publication.— Time to publication was calculated by subtracting the year funding began from the year of publication of the first original research article on the health effects of passive smoking resulting from the grant. Unpublished studies were censored at the year the interview was conducted. The funding start date was used to measure time to publication since the completion date was difficult to define.

Statistical Significance of Data.— The statistical significance of each study was classified as "statistically significant" (results indicate a statistically significant association between passive smoking and a harmful health effect); "statistically nonsignificant" (results do not indicate a statistically significant association); or "mixed" (multiple primary outcomes were measured and at least 1 was statistically significant). "Statistically significant" was defined as a P ≤.05, an odds ratio not overlapping 1, or other statistics typically characterized as significant. For unpublished studies, the categorization of statistical significance was based on the investigator's classification of his/her unpublished results. For published studies, the categorization was based on the significance of the results presented in the first original research publication as determined by 1 of the authors (A.L.M.) and a doctorate student in epidemiology.

Study Subjects.— Classified as "animals" or "humans."

Study Design.— A study was classified as "experimental" if a test regimen initiated by the investigator was administered to a treatment group, but not a control group, or as "observational" if the study was a cohort, case-control, or descriptive study.

Study Size.— Classified as "500 or less" or "more than 500" based on the distribution of sample sizes.

Health Outcome.— Classified as "respiratory effects," "lung cancer," "cardiovascular effects," "pregnancy outcome and/or birth defects," and "other" (includes studies with mixed outcomes).

Funding Source and Length.—The primary funding source was identified by the investigator and was categorized as "government," "private," "tobacco industry," or "other" and as "external" (investigator not affiliated with funding agency) or "internal." Length of funding was classified as "5 years or less" or "more than 5 years" based on the distribution of funded studies.

Statistical Analysis

The χ2 statistic was used to compare the characteristics of participants and refusals. Predictors of time to publication were analyzed by proportional hazards models.25 When there was evidence against the proportional hazards assumptions,26P values were obtained by the nonparametric Wilcoxon test.27 Multivariate models were built using stepwise addition of variables until no further additions provided a statistically significant improvement in prediction at P =.05.

Identification of Passive Smoking Research Studies

We identified 105 investigators eligible for inclusion. Four investigators could not be located, 21 were found to be ineligible after completing the preinterview, and 2 were eliminated after being interviewed because their studies had no results on the health effects of passive smoking. The response rate among the remaining 78 investigators was 83% (65/78). Characteristics of participants and refusals are shown in Table 1. Of the 13 investigators who chose not to participate, 5 cited a lack of time, 3 felt uncomfortable participating, and 5 did not state a reason.

Table Graphic Jump LocationTable 1.—Characteristics of Participants and Refusals*

The 65 participants provided information on 84 studies. Sixty-six (79%) of these studies were identified based on the information from granting organizations and 18 (21%) were identified by the investigators.

Fifty-nine (70%) of the 84 studies had some unpublished results. The reasons stated most frequently for unpublished results were ongoing data collection or analysis (n=33 times), lack of time (n=26), and competing priorities (n=11), such as other recently funded studies, studies where passive smoking was a minor component, and career changes. Statistically nonsignificant results were cited as a reason for failure to publish for only 2 studies. One investigator stated that they chose a less prestigious journal to publish their statistically nonsignificant results. Four manuscripts (2 with statistically significant and 2 with statistically nonsignificant results) resulting from 3 studies were unpublished because they had been rejected from a journal. These manuscripts were being resubmitted.

Analysis of Time to Publication

Of the 84 identified studies, 23 were excluded from the analysis of time to publication because data analysis had not begun (n=19), the publications were published before the start of the grant (n=2), or the publications had no statistical analyses (n=2). Of the remaining 61 studies, 47 were classified as "published" and 14 as "unpublished."

Figure 1 shows that the survival curve plots for time to publication for all studies differed by the statistical significance of the study results (P=.004). The median time to publication was 5 years (95% confidence interval [CI], 4-7 years) for statistically nonsignificant studies (n=21) and 3 years (95% CI, 3-5 years) for statistically significant studies (n=33). Only 1 of the 7 studies with mixed results was published; the time to publication was 6 years.

Graphic Jump Location
Proportion of unpublished studies by statistical significance of results.

As shown in Table 2, univariate analysis of all studies showed that statistically significant results, experimental study design, study size of 500 or less, and animals as subjects were predictive of time to publication. When the studies with human participants were analyzed separately (Table 2), only the statistical significance of the results was predictive of time to publication. Multivariate analysis of all studies indicated that statistical significance (P=.001) and study design (P=.01) were the only independent predictors of time to publication (P>.16 for all other variables). Multivariate analysis of the human studies showed that only statistical significance was predictive of time to publication (P=.007) (P>.15 for all other variables).

Table Graphic Jump LocationTable 2.—Comparison of Published and Unpublished Studies*

Our results indicate that statistically significant studies are published sooner than statistically nonsignificant studies, as reported for clinical studies.19,28 This finding cannot be attributed to differences in the sample sizes, funding sources, or health outcomes measured. Our finding of a publication delay can be partially explained by the reasons that investigators cited for not publishing.

The delay in the publication of statistically nonsignificant results has implications for conducting reviews and meta-analyses. The time lag in publication argues for regular updating of reviews, as practiced by the Cochrane Collaboration,29 as statistical significance of a review could change over time. Furthermore, defining eligibility for inclusion in a review by the start date of a study could reduce bias in reviews by including unpublished results. Since 47 (77%) of the 61 studies we identified had at least some published results, researchers conducting reviews on passive smoking should be aware of these studies and seek data from them.

One limitation of our study is that we interviewed only English-speaking investigators funded by US organizations. Therefore, we may have underestimated the number of studies of the health effects of passive smoking. A second limitation is that we had to rely on principal investigators to tell us the statistical significance of their unpublished results. Finally, our ability to identify studies was limited by the inadequate record keeping and willingness to share information on the part of funding agencies or investigators. For example, only half of the tobacco-affiliated organizations responded to our multiple requests for information. Our difficulty with identifying unpublished studies supports the need to register prospectively observational studies, as has been suggested for clinical trials.30

Dickersin K, Min YI. Publication bias: the problem that won't go away.  Ann N Y Acad Sci.1993;703:135-146. Discussion 146-8.
Dickersin K. How important is publication bias? a synthesis of available data.  AIDS Educ Prev.1997;9(suppl A):15-21.
Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H. Publication bias and clinical trials.  Control Clin Trials.1987;8:343-353.
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research.  Lancet.1991;337:867-872.
Rennie D, Flanagin A. Publication bias: the triumph of hope over experience.  JAMA.1992;267:411-412.
US Environmental Protection Agency.  Respiratory Health Effects of Passive Smoking: Lung Cancer and Other Disorders . Washington, DC: Indoor Air Division, Office of Atmospheric and Indoor Air Programs, Office of Air and Radiation, US Environmental Protection Agency; 1992. No. EPA/600/6-90/006F.
US Department of Health and Human Services.  The Health Consequences of Involuntary Smoking: A Report of the Surgeon General . Washington, DC: US Dept of Health and Human Services; 1986:332. DHHS Publication (CDC) 87-8398.
Law MR, Morris JK, Wald NJ. Environmental tobacco smoke exposure and ischaemic heart disease: an evaluation of the evidence.  BMJ.1997;315:973-980.
Hackshaw AK, Law MR, Wald NJ. The accumulated evidence on lung cancer and environmental tobacco smoke.  BMJ.1997;315:980-988.
Vandenbroucke JP. Passive smoking and lung cancer: a publication bias?  Br Med J (Clin Res Ed).1988;296:391-392.
Mantel N. The passive smoking myth. In: Bieva CJ, Courtois Y, Govaerts M, eds. The Present and Future of Indoor Air Quality . Amsterdam, the Netherlands: Excerpta Medica; 1989:155-159.
Kilpatrick SJ. The epidemiology of environmental tobacco smoke (ETS) and the weight of evidence argument.  Int Surg.1992;77:131-133.
Sugita M, Kanamori M, Izuno T, Miyakawa M. Estimating a summarized odds ratio whilst eliminating publication bias in meta-analysis.  Jpn J Clin Oncol.1992;22:354-358.
Armitage AK. Environmental tobacco smoke and coronary heart disease.  J Smoking-Related Dis.1993;4:27-36.
Caldwell ADS. Give a dog-end a bad name.  J Smoking-Related Dis.1992;4:1-2.
The Tobacco Institute.  Science does not support draft documents undergoing expert review [press release]. Washington, DC: The Tobacco Institute; 1990.
Bero L, Glantz S, Rennie D. Publication bias and public health policy on environmental tobacco smoke.  JAMA.1994;272:133-136.
Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects.  BMJ.1997;315:640-645.
Ioannidis JPA. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials.  JAMA.1998;279:281-286.
Jadad A, Rennie D. The randomized controlled trial gets a middle-aged checkup.  JAMA.1998;279:319-320.
Hirayama T. Non-smoking wives of heavy smokers have a higher risk of lung cancer: a study from Japan.  Br Med J (Clin Res Ed).1981;282:183-185.
Barnes D, Bero L. Scientific quality of original research articles on environmental tobacco smoke.  Tob Control.1997;6:19-26.
Bero LA, Galbraith A, Rennie D. The publication of sponsored symposiums in medical journals.  N Engl J Med.1992;327:1135-1140.
Dickersin K, Min Y-I, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards.  JAMA.1992;267:374-378.
Cox DR. Regression models and life tables (with discussion).  J R Stat Soc B.1972;34:187-220.
Harrell FE, Lee KL. Verifying Assumptions of the Cox Proportional Hazards Model: Proceedings of the 11th Annual SAS User's Group International Conference. Cary, NC: SAS Institute Inc; 1986.
Cox DR, Oakes D. Analysis of Survival Data . London, England: Chapman & Hall; 1984:123-128.
Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects.  BMJ.1997;315:640-645.
Bero LA, Rennie D. The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care.  JAMA.1995;274:1935-1938.
Simes RJ. Publication bias: the case for an international registry of clinical trials.  J Clin Oncol.1986;4:1529-1541.

Figures

Graphic Jump Location
Proportion of unpublished studies by statistical significance of results.

Tables

Table Graphic Jump LocationTable 1.—Characteristics of Participants and Refusals*
Table Graphic Jump LocationTable 2.—Comparison of Published and Unpublished Studies*

References

Dickersin K, Min YI. Publication bias: the problem that won't go away.  Ann N Y Acad Sci.1993;703:135-146. Discussion 146-8.
Dickersin K. How important is publication bias? a synthesis of available data.  AIDS Educ Prev.1997;9(suppl A):15-21.
Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H. Publication bias and clinical trials.  Control Clin Trials.1987;8:343-353.
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research.  Lancet.1991;337:867-872.
Rennie D, Flanagin A. Publication bias: the triumph of hope over experience.  JAMA.1992;267:411-412.
US Environmental Protection Agency.  Respiratory Health Effects of Passive Smoking: Lung Cancer and Other Disorders . Washington, DC: Indoor Air Division, Office of Atmospheric and Indoor Air Programs, Office of Air and Radiation, US Environmental Protection Agency; 1992. No. EPA/600/6-90/006F.
US Department of Health and Human Services.  The Health Consequences of Involuntary Smoking: A Report of the Surgeon General . Washington, DC: US Dept of Health and Human Services; 1986:332. DHHS Publication (CDC) 87-8398.
Law MR, Morris JK, Wald NJ. Environmental tobacco smoke exposure and ischaemic heart disease: an evaluation of the evidence.  BMJ.1997;315:973-980.
Hackshaw AK, Law MR, Wald NJ. The accumulated evidence on lung cancer and environmental tobacco smoke.  BMJ.1997;315:980-988.
Vandenbroucke JP. Passive smoking and lung cancer: a publication bias?  Br Med J (Clin Res Ed).1988;296:391-392.
Mantel N. The passive smoking myth. In: Bieva CJ, Courtois Y, Govaerts M, eds. The Present and Future of Indoor Air Quality . Amsterdam, the Netherlands: Excerpta Medica; 1989:155-159.
Kilpatrick SJ. The epidemiology of environmental tobacco smoke (ETS) and the weight of evidence argument.  Int Surg.1992;77:131-133.
Sugita M, Kanamori M, Izuno T, Miyakawa M. Estimating a summarized odds ratio whilst eliminating publication bias in meta-analysis.  Jpn J Clin Oncol.1992;22:354-358.
Armitage AK. Environmental tobacco smoke and coronary heart disease.  J Smoking-Related Dis.1993;4:27-36.
Caldwell ADS. Give a dog-end a bad name.  J Smoking-Related Dis.1992;4:1-2.
The Tobacco Institute.  Science does not support draft documents undergoing expert review [press release]. Washington, DC: The Tobacco Institute; 1990.
Bero L, Glantz S, Rennie D. Publication bias and public health policy on environmental tobacco smoke.  JAMA.1994;272:133-136.
Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects.  BMJ.1997;315:640-645.
Ioannidis JPA. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials.  JAMA.1998;279:281-286.
Jadad A, Rennie D. The randomized controlled trial gets a middle-aged checkup.  JAMA.1998;279:319-320.
Hirayama T. Non-smoking wives of heavy smokers have a higher risk of lung cancer: a study from Japan.  Br Med J (Clin Res Ed).1981;282:183-185.
Barnes D, Bero L. Scientific quality of original research articles on environmental tobacco smoke.  Tob Control.1997;6:19-26.
Bero LA, Galbraith A, Rennie D. The publication of sponsored symposiums in medical journals.  N Engl J Med.1992;327:1135-1140.
Dickersin K, Min Y-I, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards.  JAMA.1992;267:374-378.
Cox DR. Regression models and life tables (with discussion).  J R Stat Soc B.1972;34:187-220.
Harrell FE, Lee KL. Verifying Assumptions of the Cox Proportional Hazards Model: Proceedings of the 11th Annual SAS User's Group International Conference. Cary, NC: SAS Institute Inc; 1986.
Cox DR, Oakes D. Analysis of Survival Data . London, England: Chapman & Hall; 1984:123-128.
Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects.  BMJ.1997;315:640-645.
Bero LA, Rennie D. The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care.  JAMA.1995;274:1935-1938.
Simes RJ. Publication bias: the case for an international registry of clinical trials.  J Clin Oncol.1986;4:1529-1541.
CME
Also Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
Please click the checkbox indicating that you have read the full article in order to submit your answers.
Your answers have been saved for later.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

1,142 Views
79 Citations

Related Content

Customize your page view by dragging & repositioning the boxes below.

×