Why Review Articles on the Health Effects of Passive Smoking Reach Different Conclusions FREE

THE US Environmental Protection Agency (EPA),¹ the US Surgeon General,² the National Research Council/National Academy of Sciences,³ and the International Agency for Research on Cancer⁴ have all reviewed the scientific evidence regarding the health effects of exposure to environmental tobacco smoke, and they have all concluded that passive smoking increases the risk of diseases such as lung cancer in adults and respiratory disorders in children. Several more recent reviews have found that passive smoking is also associated with an increased risk of heart disease^{5 - 6} and with sudden infant death syndrome.⁷ However, many review articles published in the scientific literature have concluded that passive smoking is not harmful to health.⁸ Given that a clinician might rely on a single review article to provide an accurate and up-to-date overview on a topic of interest, it is somewhat disconcerting that not all published review articles are reaching the same conclusion about the health effects of passive smoking, particularly when there is consensus in the scientific community that passive smoking is harmful. The goal of this study was to identify factors that might explain why review articles on the health effects of passive smoking are reaching different conclusions.

Several interrelated factors may influence the conclusions of review articles. First, the conclusions of review articles may vary depending on the quality of the review conducted. Review article quality is generally measured in terms of the degree to which a systematic strategy has been used to evaluate the evidence on a particular topic.^{9 - 15} It is generally believed that reviews that have been conducted systematically are less likely to reach biased conclusions than those that have not.^{9 ,11 ,15 - 16} For example, Antman et al⁹ found that, for reviews of myocardial infarction treatment, the conclusions of unsystematic reviews were often inaccurate and out-of-date when compared with a systematic review.⁹ Therefore, in the literature on passive smoking, review articles may be reaching different conclusions depending on their quality.

Another factor that may influence the quality of review articles, and therefore their conclusions, is whether they have been subject to peer review. Research conducted by us and by others has found that, for original research articles, the quality of articles published in peer-reviewed journals is superior to the quality of articles published in non–peer-reviewed symposium proceedings.^{17 - 18} It is therefore possible that the quality of review articles also differs depending on whether they have been peer reviewed, and this difference in quality may lead to differences in conclusions.

Article conclusions may also be influenced by sources of funding or author affiliations. Several studies have found that, for a wide range of industries, publications based on industry-funded research tend to draw pro-industry conclusions.^{19 - 24} Similarly, studies have found that authors of review articles tend to draw conclusions that are consistent with their professional affiliations.^{25 - 26} In the literature on passive smoking, it is therefore plausible that review article conclusions would be associated with tobacco industry sponsorship or tobacco industry–affiliated authors.

The specific topic of a review may also play a role in what conclusion is reached. For example, it is possible that passive smoking is truly associated with lung cancer and heart disease in adults, but not with brain tumors in children. One would therefore expect review articles on these different topics to reach different conclusions about the health effects of passive smoking.

Finally, one might expect the year of publication of a review article to be associated with its conclusion. For example, a review article published during the mid 1980s would be based on fewer original research articles than one published today, and it therefore might be less likely to reach a firm conclusion about the health effects of passive smoking.

The goal of our study was to evaluate the quality of review articles on the health effects of passive smoking and to determine whether the conclusions of review articles are primarily associated with their quality or with other article characteristics. Our a priori hypotheses were that review articles concluding that passive smoking is not harmful would tend to be poor in quality, published in non–peer-reviewed symposium proceedings, and written by investigators with tobacco industry affiliations. We also examined the topic of the review and the year of publication as potential confounding factors.

Review articles on the health effects of passive smoking were identified by searching MEDLINE and EMBASE from 1980 through 1995 using a variety of key words and subject headings related to passive smoking and review and meta-analysis. The search strategy was developed in consultation with a librarian and was based on the strategy used by the Cochrane Collaboration, an international group dedicated to conducting systematic reviews of the biomedical literature.²⁷ Additional review articles were identified from a database of symposium articles on passive smoking that had been gathered for a previous study.⁸ These articles were originally identified by searching MEDLINE, CATALOG, DIALOG, Conference Papers Index, TOXLINE, and International Guide to Periodicals from 1965 through 1993 for symposium proceedings related to passive smoking; in addition, 2 symposia were identified through Tobacco Institute press releases.

An article was included if it met the following criteria: (1) its stated or implied purpose was to review the scientific evidence suggesting that passive smoking is associated with 1 or more health outcomes; (2) it focused specifically on the health effects of passive smoking, rather than reviewing several causes of a particular disease; (3) it was written in English; and (4) it was published between 1980 and 1995. An article was excluded if it reviewed aspects of passive smoking other than health, such as exposure assessment or policy issues; if it discussed several different risk factors for disease, rather than focusing on the effects of passive smoking; or if it was an editorial, commentary, or letter to the editor. A total of 106 articles that satisfied these inclusion and exclusion criteria were identified.

We hired 2 independent assessors, both of whom had experience conducting systematic reviews, to evaluate the quality of the review articles identified. Assessors were trained to use our quality assessment instrument and were provided with a comprehensive set of instructions for use during the study. Quality assessors were blinded to our study hypotheses and were told that the sole purpose of the study was to evaluate the quality of review articles on the health effects of passive smoking. In addition, assessors were blinded to all identifying characteristics of the articles: author names and affiliations, journal titles, acknowledgments, and dates of publication were removed completely. Articles were sent to assessors in a random order using a random number generator on a computer. Assessors stated that they had never been affiliated in any way with either the tobacco industry or tobacco control groups and that they had not previously reviewed the literature on passive smoking.

Review article quality was evaluated using a slightly modified version of the Oxman instrument, which is the only instrument available for assessing review article quality that has been published and tested for validity and reliability.^{28 - 29} The criteria used to evaluate review article quality are listed in Table 1: items 2 through 9 and item 12 were based on the Oxman instrument. Items 1, 10, and 11 were added because they have been used by other researchers to evaluate the quality of reviews.^{10 - 11 ,25 ,30}

For each criterion, the quality assessor could answer yes (2 points), partial (1 point), no (0 points), or can't tell (0 points), and the quality score was the number of points awarded divided by 24, the total number of points possible. The quality score could therefore have ranged from 0 (lowest) to 1 (highest).

Analyses were based on the mean quality score (the average of the 2 assessors' scores). If the assessors' scores differed by more than 1 SD (0.20 point), they were asked to discuss their answers until they achieved consensus, and the consensus score was used. Seventeen percent (18/106) of the articles were reevaluated using this consensus process. Agreement between reviewers was evaluated using the method proposed by Bland and Altman.³¹ The median difference between reviewers' scores was 0, and the 2.5%, 25%, 75%, and 97.5% quantiles were −0.21, −0.04, 0.13, and 0.21, respectively. This means that 95% of the time, reviewers' scores were within 0.21 of each other (approximately 1 SD), on a scale from 0 to 1. The correlation between the 2 reviewers' scores was 0.87.

Articles were classified as concluding that passive smoking is either harmful or not harmful. The conclusion was classified as harmful if the authors stated that passive smoking was definitely or probably harmful. The conclusion was classified as not harmful if the authors stated that passive smoking was definitely or probably not harmful, that the statistical association between passive smoking and adverse health outcomes was attributable to poor study design or bias, or that the evidence was inconclusive. Articles concluding that the evidence was inconclusive were classified in the not harmful category because this conclusion is consistent with an acceptance of the null hypothesis that there is no relationship between passive smoking and disease.

Article conclusions were classified independently by the 2 quality assessors, who were blinded to study hypotheses and all identifying aspects of the articles, and by one of us (D.E.B.), who was blinded to article quality scores. Interrater agreement was excellent: all 3 raters agreed on the conclusion category for 89% (94/106) of the articles (overall κ=0.83). When all 3 raters did not agree on the article conclusion category, the category assigned by the majority was used.

Other descriptive characteristics of the articles were classified by one of us (D.E.B.), who was blinded to article quality scores. If the classification was unclear, the principal investigator (L.A.B.) was consulted.

The peer review status was classified as either peer reviewed or non–peer reviewed based on statements in the parent publication. A publication was considered peer reviewed if it stated that it was peer reviewed, if it published a list of peer reviewers, or if it required that multiple manuscript copies be submitted for review prior to publication; otherwise, it was considered non–peer reviewed. Peer review status was classified as missing for 3 articles for which we were unable to obtain the publications' instructions for authors. These articles were automatically dropped from multivariate analyses in which the peer review status variable was included in the model.

Of the 106 articles in our study, 77% failed to disclose the sources of funding for the research. Therefore, we assessed potential financial conflicts of interest by classifying the authors of the articles as either tobacco industry affiliated or non–tobacco industry affiliated. An article was classified as having tobacco industry–affiliated authors if 1 or more of the authors had ever (1) received funding from a tobacco company or the Tobacco Institute, based on acknowledgments in articles gathered for this study and for a prior study of the literature on passive smoking¹⁸ ; (2) received funding from the tobacco industry–financed Council for Tobacco Research special projects division, based on a published list³² ; (3) received funding from the tobacco industry–financed Center for Indoor Air Research (CIAR) for a special-reviewed project, based on information gathered from CIAR for a prior study¹⁹ ; (4) submitted a statement on behalf of the tobacco industry regarding the EPA's risk assessment on passive smoking, based on information gathered from the EPA for a prior study³³ ; or (5) had participated in at least 2 tobacco industry–sponsored symposia, based on information gathered for a prior study.⁸ Otherwise, the article was classified as having non–tobacco-affiliated authors.

The article topic was classified as lung cancer, heart disease, respiratory disease, multiple health outcomes, or other health effect.

The year of publication was analyzed both continuously and categorically as 1980-1986, 1987-1991, or 1992-1995. These categorizations were used because the surgeon general and the National Academy of Sciences both published consensus reports on the health effects of passive smoking in 1986, and the EPA published its risk assessment of passive smoking early in 1992, and we hypothesized that the conclusions of review articles in the scientific literature might be influenced by publication of these landmark documents.

To compare mean quality scores in various groups we used t tests and analysis of variance. Associations between categorical variables were evaluated using χ² analyses. Multiple logistic regression was used to determine which article characteristics were most highly associated with concluding that passive smoking is not harmful to health. The predictor variables were article quality score (analyzed as a continuous variable), peer review status, author affiliation, article topic, and year of publication (examined as both a continuous variable and a categorical variable). Multilevel categorical variables were modeled using indicator variables. Sensitivity analyses and diagnostic tests were performed to evaluate the multiple logistic regression model. Two-tailed P values less than .05 were considered statistically significant in all tests.

Descriptive characteristics of the review articles identified are presented in Table 2. Overall, 37% of articles concluded that passive smoking is not harmful. Most reviews were unsystematic but had been peer reviewed.

Thirty-one review articles (29%) were written by authors with tobacco industry affiliations. For 30 of the 31 articles, the author was affiliated with the tobacco industry either prior to or concurrent with the year of publication of the review. For the 1 review that was the exception, the article was published during the year prior to the first documented affiliation.

Table 1 presents our results related to the quality of review articles. The mean quality score was 0.36 (SD, 0.20; range, 0.04-0.94). This means that the average review article in our study satisfied only one third of the criteria on our quality assessment instrument.

Table 3 shows that there was a strong relationship between the conclusion of a review and the affiliation of its authors. Ninety-four percent (29/31) of reviews by tobacco industry–affiliated authors concluded that passive smoking is not harmful, compared with 13% (10/75) of reviews by authors without tobacco industry affiliations (P<.001). In our study, the relative risk (RR) of concluding that passive smoking is not harmful, comparing tobacco industry–affiliated authors with nonaffiliated authors, was 7.0 (95% confidence interval [CI], 3.9-12.6). The corresponding odds ratio (OR) was 94.2 (95% CI, 19.4-457.6). These measures are dissimilar because the outcome (concluding that passive smoking is not harmful) was not rare among the group of tobacco industry–affiliated authors.

The results of the full logistic regression model are presented in Table 4. The odds that a review article with tobacco industry–affiliated authors would conclude that passive smoking is not harmful were 88.4 times higher than the odds for a review article with non–tobacco-affiliated authors, when controlling for article quality, peer review status, article topic, and year of publication (95% CI, 16.4-476.5; P<.001).

We conducted sensitivity analyses to determine whether this finding was limited to a particular subset of articles in our sample. When we limited the analysis to higher-quality articles (defined as articles receiving mean quality scores of 0.50 or higher), tobacco industry affiliation remained the only factor associated with concluding that passive smoking is not harmful (OR, 85; 95% CI, 3-2134; P=.007). (The article topic variable was dropped from this analysis because of small cell sizes that produced instability in the model.) Similarly, tobacco industry affiliation was the only factor associated with concluding that passive smoking is not harmful when the analysis was restricted to peer-reviewed articles only (OR, 93; 95% CI, 9-945; P<.001). No matter how we analyzed the data, tobacco industry affiliation was the only factor associated with concluding that passive smoking is not harmful to health in the multivariate analyses.

We also stratified the analysis by author affiliation to determine whether, within either the group of articles by tobacco industry–affiliated authors or the group of articles by non–tobacco-affiliated authors, review article conclusions might be associated with other factors. However, within each group, we found no other factors that were significantly predictive of review article conclusions.

To determine whether our results had been influenced by classification of reviews with "inconclusive" findings in the "passive smoking not harmful" category, we reanalyzed the data after excluding inconclusive reviews. This resulted in exclusion of 24 reviews (17 by tobacco-affiliated authors, 7 by nonaffiliated authors). The magnitude of the association between author affiliation and review article conclusion was stronger in this analysis (OR, 130; 95% CI, 20-862; P<.001).

Finally, we conducted diagnostic tests to determine whether some articles were highly influential in our analyses. We found that 2 articles by tobacco industry–affiliated authors that concluded passive smoking is harmful were highly influential; however, when these 2 articles were excluded from the analysis, 100% of the reviews by tobacco industry–affiliated authors concluded that passive smoking is not harmful.

The goal of our study was to determine whether review article quality, author affiliation, or other article characteristics were associated with concluding that passive smoking is not harmful to health. We had initially hypothesized that review articles would be more likely to conclude that passive smoking is not harmful if they were poor in quality, published in non–peer-reviewed journals or symposium proceedings, or written by authors with tobacco industry affiliations. We also speculated that the topic of a review and its year of publication might be associated with its conclusion.

We found that very few reviews had been conducted systematically, resulting in overall mean quality scores that were quite low. The low quality scores may be attributable in part to poor reporting in some of the articles; for example, several authors implied that they had conducted a review of the literature by making statements such as "there are currently x number of published studies on a particular topic." These results highlight the need for accurate reporting of study methods in reviews as well as original research articles.

The quality of a review was not associated with its conclusion when controlling for the effect of author affiliation. Using multivariate logistic regression analysis, the only factor that predicted a review article's conclusion was whether its author was affiliated with the tobacco industry. This finding was consistent and emerged no matter how we analyzed the data.

A total of 10 reviews^{34 - 43} by authors classified as non–industry affiliated concluded that passive smoking may not be harmful. In 7 cases,^{34 - 40} the authors found that the evidence was inconclusive. For example, 1 review³⁷ stated that "[w]hile a few well-designed studies demonstrate a significant effect of passive smoking on child health, most studies had significant design problems that prevent reliance on their conclusions." Two of the other 3 reviews^{42 - 43} had authors who had some affiliation with the tobacco industry, although they did not meet the stringent criteria used to define affiliation in this study; in both cases, the authors had participated in a single tobacco industry–sponsored symposium prior to or concurrent with publication of the review. Therefore, we identified only 1 review⁴⁰ written by an author without any known tobacco industry affiliations that concluded that passive smoking is not harmful to health.

Our findings suggest that the discrepancy between consensus documents and published reviews related to the health hazards of passive smoking is primarily attributable to large numbers of reviews written by authors with tobacco industry affiliations. For example, although 37% (39/106) of reviews in our study concluded that passive smoking is not harmful, 74% (29/39) of these were written by authors with tobacco industry affiliations. From the time our search ended in 1995, at least 17 reviews of the health effects of passive smoking have been published. Two of the most recent reviews by authors not affiliated with the tobacco industry conclude that passive smoking is harmful.^{44 - 45}

These findings suggest that the tobacco industry may be attempting to influence scientific opinion by flooding the scientific literature with large numbers of review articles supporting its position that passive smoking is not harmful to health. This conclusion is consistent with the industry's previous strategies related to tobacco. For example, internal documents have shown that one of the tobacco industry's key strategies has been to suggest that there is doubt or controversy about scientific knowledge related to the health effects of tobacco.^{46 - 48} In this way, the industry is able to argue that government regulations are not warranted.

Our findings are unlikely to be attributable to bias. Review articles were identified using a systematic strategy and well-defined inclusion and exclusion criteria. Article quality was evaluated by independent assessors who were trained, used a slightly modified version of a valid and reliable instrument, and were blinded to study hypotheses. In addition, article conclusions were classified independently by 3 people, 2 of whom were blinded to all identifying aspects of the articles, including author names and affiliations. Furthermore, we do not feel that our findings were biased by inclusion of symposium studies that we had identified for a prior study, because our results were the same when these articles were excluded.

Our findings are consistent with previous research on both the quality and conclusions of review articles. Several studies have found that most published reviews are unsystematic and that their quality is therefore low.^{10 - 11 ,13 ,25} In addition, other studies have found an association between the conclusions of review articles and the affiliations of their authors. For example, Assendelft et al²⁵ found that reviews were more likely to conclude that spinal manipulation was beneficial if one of the authors was a spinal manipulator. Similarly, Chalmers et al²⁶ showed that, for several types of controversial procedures, an author's enthusiasm for the procedure was associated with his or her specialty. Furthermore, several investigators have found that original research articles that acknowledge sponsorship from the pharmaceutical industry,^{20 - 21 ,23} the chemical industry,²⁴ or the tobacco industry¹⁹ tend to draw pro-industry conclusions. Ultimately, the conclusion of any review article must be based on the judgment and interpretation of the author.

Because research studies on a variety of topics have consistently found an association between the affiliations of an author and the conclusions of his or her published research, we feel that our findings may be generalizable to review articles on topics other than passive smoking. That is, the conclusions of a review article may be suspect whenever the author has a financial interest in the outcome of the review. Therefore, our findings suggest that the authors of review articles should disclose their affiliations, sources of funding, and other potential financial conflicts of interest, and that the readers of review articles should consider these disclosures when deciding how to judge an article's conclusions.