Journal Prestige, Publication Bias, and Other Characteristics Associated With Citation of Published Studies in Peer-Reviewed Journals FREE

Although publication is a crucial portion of the scientific process, an equally important part is the subsequent use and citation of these published articles by other researchers and authors. We studied a cohort of all research submitted to a scientific meeting and subsequently published to determine how these studies were cited by other authors and determine what characteristics (including positive results) were associated with more frequent citation.

We previously reported the methods of the first phase of this study.¹ To summarize, all abstracts of scientific studies submitted to the Society for Academic Emergency Medicine (SAEM) meeting in 1991 were examined. Each submitted abstract was categorized in a blinded fashion according to research design, number of subjects, and other characteristics (Table 1 and Table 2), and rated subjectively for scientific quality and newsworthiness using a modified delphi method. Searches of MEDLINE, EMBASE, and Cochrane databases were conducted for 4 years after the meeting to identify publication in any journal listed in the National Library of Medicine (NLM). If needed, further information was obtained from authors in writing.² The characteristics of submitted abstracts that predicted full publication were reported previously.¹

In March 2000, the cohort was further examined by searching the Science Citation Index (SCI) database for articles of every study submitted to this meeting that had been published in full (http://www.webofscience.com). For each published article, all citations of that article from publication to the time of the search were identified. Citations were minimal until 2 years after publication, so results were analyzed beginning then and for the next 3.5 years.

Two outcome measures were determined. The number of citations per year during the study period was calculated for each article. This can be considered the article's "impact factor," analogous to the traditional impact factor for journals (the annual number of citations per published article in a journal).³ The second outcome measure was the mean citing journal impact factor (CJIF) of each published article, calculated by averaging the impact factors of each journal citing that publicaton. Together, these 2 outcomes provide estimates of both the quantity and quality of citations of a manuscript.

Summary statistics were calculated using Systat 9.02 statistical software (SPSS, Evanston, Ill). The relationship of the primary outcome measures (citations per year and CJIF) to characteristics of the original abstracts was assessed using regression trees (CART, or Classification and Regression Trees).⁴ Exploratory analyses of the relationships of interest in this study suggested that simpler multiple regression models would not fit the data as well as regression trees, which are particularly suited to complex interactions among predictors.

The analysis was conducted using CART 4.0 (Salford Systems, San Diego, Calif). The candidate predictor variables were impact factor of the publishing journal, number of subjects (in quartiles, 0-33, 34-101, 102-425, and >425), subjects (human, animal, or other), binary variables representing presence or absence of an explicit hypothesis, control group, blinding, acceptance for presentation at the research meeting, method (prospective or retrospective), and positive results (as previously defined¹), and the subjective "newsworthiness" and quality scores, derived from the full article, not the abstract.

The model was allowed to continue splitting until a node contained 20 cases or fewer (about 10% of the sample) and the resultant trees were tested by 10-fold cross-validation analysis to eliminate over-fit trees and to identify the best model. The cross-validation relative error was used as a pseudo-R₂ measure (analogous to but not truly R₂), a global measure of the final model's explanatory power on a new data set (a more detailed description of the methodology is available from the authors).

This study was approved by the committee on human research of the University of California.

Four hundred ninety-three studies were submitted to the 1991 SAEM meeting from a total of 144 institutions and 103 medical schools, of which 179 (36%) were accepted for presentation at the meeting. Two hundred nineteen (44%) of the 493 studies submitted to this meeting were published in 44 peer-reviewed journals (37 in specialties other than emergency medicine) with impact factors ranging from 0.23 to 24.5.¹

Fifteen of the published articles appeared in journals that are not SCI citation sources and thus have no SCI impact factor. These were excluded from subsequent analysis, leaving 204 published articles.

Of the 204 included published articles, 19 (9.3%) had no citations during the study period; the remainder were cited a total of 1446 times by 440 different journals with impact factors ranging from 0.01 to 24.5, 434 of them from disciplines other than emergency medicine. Seventy-nine studies (39%) were cited 1 or 2 times only during the study period. The mean citations per year was 2.04 (95% confidence interval [CI], 1.6-2.4; range, 0-20.9) and the mean impact factor of the citing journals was 1.69 (95% CI, 1.50-1.87; range, 0.01-9.9). Citing journals included all the large general medical journals; 18% of citing journals had an impact factor greater than 3, compared with only 10% of all journals in the SCI.⁵

Univariate analysis is not reported for purposes of brevity. CART regression for citations per year yielded a pseudo-R² statistic of 0.14, suggesting relatively low explanatory power. The regression tree suggested that impact factor was the only variable of importance (Table 1); other characteristics of the studies either had no influence, or were almost completely subsumed by the impact factor of the publishing journal. After adjustment for the impact factor of the publishing journal, presence of a control group, the subjective "newsworthiness" score, and sample size were the next most important determinants of citation (Table 1).

The regression analysis for CJIF yielded a pseudo-R² statistic of 0.09, also suggesting little predictive ability. The impact factor of the publishing journal was the most important variable here also; however, in contrast to citations per year, the newsworthiness and quality scores also contributed substantially (Table 2).

Positive outcome bias was not evident in this sample in either univariate analysis or the regression model. Full manuscripts with negative outcomes had a mean 1.96 citations per year (95% CI, 1.2-2.7), whereas those with positive outcomes had 1.84 citations per year (95% CI, 1.2-2.5). Results for CJIF were similar.

The publication of research in peer reviewed journals is only an intermediate outcome, satisfying to authors but not necessarily useful to others. There is no way to measure how useful a published article is to clinicians, but we can measure its impact on other authors by how frequently they cite it in their publications. Such citations are a hallmark of academic achievement for authors and for journals, correlate highly with the opinion of peers as to a scientist's contribution to his/her field and are used by medical school deans for promotion reviews.^6- 7 Citations complete the chain of publication and that underpins the evolution of scientific knowledge.

Of the 204 publications studied, 185 were cited a total of 1446 times by 440 different journals. The average article was cited 2.04 times per year, and the mean citing journal impact factor was 1.84. This is roughly comparable to the citation rate of all material published in all journals in the SCI,⁸ and the proportion of citing journals with high impact factors in our sample was greater than that of the SCI as a whole.⁵

Only 19 (9.3%) of these studies were never cited during the study period (even by their own authors), compared with 22% for all the international medical literature,⁹ 11% of AIDS articles, and 15% of biology articles.¹⁰ These figures suggest that the study cohort was broadly representative of the biomedical literature.

We found that the impact factor of the original publishing journal, not the methodology or quality of the research, was the strongest predictor of citations per year. A pessimist would suggest that despite the era of accessibility due to electronic searching and retrieval, citation may be more strongly influenced by the reputation of the publishing journal than by the design merits of the study. Thus, a strong or seminal paper submitted to a minor journal might not receive the scientific recognition it deserves. Likewise, a weak article published in a major journal will probably receive more recognition than it deserves.

An optimist might interpret our results to mean that journals are perfectly efficient in publishing studies of a uniform quality, with the same citation value as the journal itself. However, the SCI reports marked variability in citation rates of individual articles in any given journal,⁵ and we found no relationship between study design (and other measures of quality), and the impact factor of the original publishing journal.¹

Once the impact factor of the publishing journal was accounted for, the subjective newsworthiness score (from a delphi panel rating), sample size and presence of a control group were the only major predictors of citations (and whose respective contribution was only 26.0%, 26.5%, and 24.3% as strongly). This is disappointing since one would hope that characteristics of sound design would be the most important predictors. However, it may also indicate that quality checklists for assessing studies are not meaningful or accurate^11- 12 or that editors are able to identify importance and originality in research independent of the "plumbing" of study design.

It is encouraging that studies with positive results were not cited more frequently or by more prestigious journals. This is in contrast to the consistent bias toward acceptance and publication of studies with positive results previously reported for this study cohort and others.^1,13- 20 Although documented at earlier steps of the publication process (from submission to research meetings to publication of full articles), the final step, citation by subsequent authors, appears to be relatively free of this bias.

In this study, acceptance of research for presentation at the meeting had no predictive power for citation frequency or impact factor of citing journals. As with our previous finding that acceptance failed to predict publication of a manuscript or the impact factor of the publishing journal,¹ the current results suggest methods of screening research for meeting presentation are very fallible.

Our study has several limitations. It examined research (mostly clinical) from only 1 meeting although this cohort has previously been shown to share key characteristics with research from other specialty meetings¹ and was published in 44 journals covering a broad spectrum of specialties and impact factors. Eighty-four percent of publishing journals and 98% of the citing journals were from specialties other than that sponsoring the meeting. Citations per year is not an ideal tool for detecting positive outcome bias because a citation may not credit or praise a study but instead refute or criticize it. However, most citations in most articles do not refute the cited paper, and authors, journals, and promotion committees consider the number of citations to constitute a measure of merit. Finally, the ability of the regression model to predict outcomes was poor, meaning that most of the variance was accounted for by unmeasured variables (such as, hopefully, the relevance of the study to the citing research).