0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Peer Review Congress |

US and Non-US Submissions:  An Analysis of Reviewer Bias FREE

Ann M. Link, MA
[+] Author Affiliations

From Gastroenterology , Bethesda, Md. Ms Link is now with the Journal of Immunology , Bethesda, Md.


JAMA. 1998;280(3):246-247. doi:10.1001/jama.280.3.246.
Text Size: A A A
Published online

Context.— Reviewers increasingly are asked to review manuscripts from outside their own country, but whether they are more likely to recommend acceptance of such manuscripts is not known.

Objective.— To assess whether US reviewers or non-US reviewers evaluate manuscripts differently, depending on whether the manuscripts are submitted from outside the United States or from the United States.

Design and Setting.— A retrospective analysis of all original submissions received by Gastroenterology in 1995 and 1996. Reviewers ranked manuscripts in 4 decision categories: accept, provisionally accept, reject with resubmission, or reject.

Main Outcome Measure.— Ranking of papers based on nationality of authors and reviewers.

Results.— The percentage of non-US manuscripts placed in each decision category by US (n=2355) and non-US reviewers (n=1297) was nearly identical (P=.31). However, US reviewers recommended acceptance of papers submitted by US authors more often than did non-US reviewers (P=.001). Non-US reviewers ranked US papers slightly more favorably than non-US papers (P=.09), while US reviewers ranked US papers much more favorably (P=.001).

Conclusions.— Reviewers from the United States and outside the United States evaluate non-US papers similarly and evaluate papers submitted by US authors more favorably, with US reviewers having a significant preference for US papers.

MANY US journals cite increased numbers of international submissions.13 Seventy percent of submissions to Gastroenterology are from outside the United States. However, non-US submissions are less likely to be published based on peer review. These data prompted a study assessing whether US and non-US reviewers evaluate manuscripts submitted to Gastroenterology from outside the US differently than manuscripts submitted from US-based authors.

Using a manuscript tracking database, a retrospective study was conducted of original research articles submitted to Gastroenterology during 1995 and 1996. Manuscripts not sent out for peer review and revised manuscripts were excluded. The corresponding author's nationality, each reviewer's nationality, and each reviewer's ranking of the paper were obtained. Reviewers who did not indicate a decision ranking were excluded. Decision rankings of papers were obtained from the reviewer forms. These forms ask reviewers to recommend one of the following decisions on a paper: accept, provisionally accept, reject with resubmission, or reject. Two nonblinded reviewers are asked to review each paper.

Authors and reviewers were identified as either "US" or "non-US." Reviews were tabulated according to author nationality, reviewer nationality, and reviewer ranking combinations. Comparisons of reviewer recommendations and reviewers' nationality were assessed using the χ2 test for independence (with 3 df). Reviewer recommendations and origin of the paper were analyzed in the same manner.

International papers were assessed to determine if they were ranked differently by US and non-US reviewers. There were 2355 US reviewers and 1297 non-US reviewers of non-US papers. The percentages of international papers placed in each decision category by US and non-US reviewers were similar, indicating no significant correlation between a reviewers' nationality and the ranking of international papers (accept: 3.6% vs 3.2%; provisionally accept: 24.9% vs 24.7%; reject with resubmission: 26.8% vs 24.6%; reject: 44.7% vs 47.6%) (P=.31).

Next, US papers were analyzed to determine if there were differences in US and non-US reviewers' rankings. There were 1174 US reviewers and 449 non-US reviewers of US papers. US reviewers placed papers in the 3 acceptance categories more often than non-US reviewers (64% vs 55%) and also ranked US papers significantly higher (accept: 7.0% vs 3.6%; provisionally accept: 31.3% vs 30.5%; reject with resubmission: 26.1% vs 21.6%; reject: 35.6% vs 44.3%) (P=.001).

Data were analyzed to determine whether these findings indicated bias. Reviewers' evaluations were examined to determine whether US reviewers evaluated non-US and US papers similarly. This process was also used to determine whether non-US reviewers evaluate the 2 sets of papers similarly. Non-US reviewers rank papers submitted from the United States more favorably (accept: 3.6% vs 3.2%; provisionally accept: 30.5% vs 24.7%; reject with resubmission: 21.6% vs 24.6%; reject: 44.3% vs 47.6%; P=.09). Moreover, US reviewers show a stronger preference for US papers than non-US reviewers (accept: 7.0% vs 3.6%; provisionally accept: 31.3% vs 24.9%; reject with resubmission: 26.1% vs 26.8%; reject: 35.6% vs 44.7%; P=.001).

Based on logistic regression looking simultaneously at reviewer nationality and manuscript source, the data show that the manuscript source was significant (P=.001), with domestic papers having an odds ratio of 1.49 for background of review and background of acceptance, while the reviewer's nationality was not significant (P=.22) (Table 1).

Table Graphic Jump LocationAcceptance Rates Based on Dichotomization

Several factors prevent conclusive findings on international bias in this study. First, non-US reviewers were pooled because there were not enough authors and reviewers from the same country to obtain statistically significant results. Second, nationality was based on the corresponding author's location rather than actual nationality. Third, associate editors may choose non-US reveiwers who have similiar training and viewpoints; however, data regarding location of training were not available. Fourth, reviewers were not blinded. Most important, bias cannot be unequivocally inferred without measuring the quality of individual papers. Still, this study does provide insights into potential variability in manuscript evaluations across national lines and lends itself to further investigation and more complex studies.

Stossel TP, Stossel S. Declining American representation in leading clinical research journals.  N Engl J Med.1990;322:739-742.
LaRusso NF, Link AM. Gastroenterology: an international journal.  Gastroenterology.1995;108:625-626.
Berk PD. Hepatology and hepatology: the good, the bad, and the ugly.  Hepatology.1996;24:1530-1532.

Figures

Tables

Table Graphic Jump LocationAcceptance Rates Based on Dichotomization

References

Stossel TP, Stossel S. Declining American representation in leading clinical research journals.  N Engl J Med.1990;322:739-742.
LaRusso NF, Link AM. Gastroenterology: an international journal.  Gastroenterology.1995;108:625-626.
Berk PD. Hepatology and hepatology: the good, the bad, and the ugly.  Hepatology.1996;24:1530-1532.

Letters

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 100

Related Content

Customize your page view by dragging & repositioning the boxes below.