0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Brief Report |

Differences in Review Quality and Recommendations for Publication Between Peer Reviewers Suggested by Authors or by Editors FREE

Sara Schroter, PhD; Leanne Tite, MSc; Andrew Hutchings, MSc; Nick Black, MD
[+] Author Affiliations

Author Affiliations:BMJ Editorial Office, BMA House, London (Dr Schroter and Ms Tite); Health Services Research Unit, London School of Hygiene & Tropical Medicine, London (Mr Hutchings and Dr Black), England.

More Author Information
JAMA. 2006;295(3):314-317. doi:10.1001/jama.295.3.314.
Text Size: A A A
Published online

Context Many journals give authors who submit papers the opportunity to suggest reviewers. Use of these reviewers varies by journal and little is known about the quality of the reviews they produce.

Objective To compare author- and editor-suggested reviewers to investigate differences in review quality and recommendations for publication.

Design, Setting, and Participants Observational study of original research papers sent for external review at 10 biomedical journals. Editors were instructed to make decisions about their choice of reviewers in their usual manner. Journal administrators then requested additional reviews from the author's list of suggestions according to a strict protocol.

Main Outcome Measure Review quality using the Review Quality Instrument and the proportion of reviewers recommending acceptance (including minor revision), revision, or rejection.

Results There were 788 reviews for 329 manuscripts. Review quality (mean difference in Review Quality Instrument score, −0.05; P = .27) did not differ significantly between author- and editor-suggested reviewers. The author-suggested reviewers were more likely to recommend acceptance (odds ratio, 1.64; 95% confidence interval, 1.02-2.66) or revise (odds ratio, 2.66; 95% confidence interval, 1.43-4.97). This difference was larger in the open reviews of BMJ than among the blinded reviews of other journals for acceptance (P = .02). Where author- and editor-suggested reviewers differed in their recommendations, the final editorial decision to accept or reject a study was evenly balanced (50.9% of decisions consistent with the preferences of the author-suggested reviewers).

Conclusions Author- and editor-suggested reviewers did not differ in the quality of their reviews, but author-suggested reviewers tended to make more favorable recommendations for publication. Editors can be confident that reviewers suggested by authors will complete adequate reviews of manuscripts, but should be cautious about relying on their recommendations for publication.

Figures in this Article

Peer review plays a central role in determining what research is published. Peer reviewers are responsible for identifying methodological flaws and for improving the quality of manuscripts. Several factors are associated with review quality (reviewer age, being a current investigator, and postgraduate training in epidemiology or statistics).1,2 Many journals give authors the opportunity to suggest reviewers for their own paper, but editors' decisions to select these reviewers vary because some are concerned that they might favor the author. However, many journals find it hard to recruit good-quality reviewers and, as such, are willing to try authors' suggestions.

The only study to evaluate author-suggested reviewers found that these reviewers were less critical than those suggested by editors in terms of the scientific importance of an article and the decision to publish.3 However, the generalizability of this finding is uncertain because it was based on 1 journal and did not use a validated outcome measure. We describe a large study of 10 journals across a range of medical specialties to investigate whether author-suggested reviewers differed from editor-suggested reviewers in terms of review quality and recommendation for publication.

Sample

The study was conducted of 10 journals (Table 1) that routinely request that authors suggest potential peer reviewers as part of their electronic manuscript management. Original research papers submitted between April 1, 2003, and December 31, 2003 (April 1, 2003–August 31, 2003, for BMJ) and sent out for peer review were eligible for inclusion. Papers were excluded if the author did not spontaneously suggest a reviewer, as were reviews conducted by journals' statistical reviewers.

Table Graphic Jump LocationTable 1. Manuscripts and Reviewers by Journal

We needed 92 papers with discordant recommendations between author- and editor-suggested reviewers to detect a 2-fold difference in the odds of recommendation with 90% power at 2-sided α = .05. A total of 110 papers would be sufficient to detect a difference in review quality of 0.4 (SD of difference, 1.2; 2-sided α = .05; power, 90%) on the Review Quality Instrument (RQI).4,5

Procedures

Editors chose reviewers in their usual manner. Using the journals' electronic tracking systems, administrators requested an additional review from the top of the author's list of suggestions. If the editor had already requested a review from someone on the author's list, the administrator did not request an additional review. If the first person on the list declined the review, the next reviewer on the list was contacted until a reviewer was found.

We did not seek ethics committee approval for this study because it did not involve human participants or medical records. We did not seek consent from individual reviewers because we did not interfere with the usual editorial process and reviewers were not recruited into the study. Raters of the reviews volunteered to participate and were blinded to the identity and status of the reviewer.

Outcome Measures

Review Quality. Each review was rated independently using the RQI4 (Box) by 2 of 16 trained raters who were blinded to the identity and source of the reviewer. The reliability and validity of the RQI have been reported previously.47

Recommendation to Publish. Of participating journals, 6 of 10 ask reviewers to provide a recommendation about publication. For the purpose of this study, BMJ also asked reviewers to provide a recommendation. We reclassified the journals' existing response categories as: accept (a recommendation to accept or accept with minor revisions), revise (major revisions), and reject (reject or revise and reconsider).

Papers were denoted as being preferred by author-suggested reviewers if at least 1 of the author-suggested reviewers rated the paper more favorably than the highest-rating editor-suggested reviewer, or at least 1 editor-suggested reviewer rated lower than the lowest-rating author-suggested reviewer. For editor-suggested reviewer preference the denotation was reversed. A paper could fall into both categories (eg, author-suggested reviewers recommending accept and reject but all editor-suggested reviewers recommended revise), or in situations in which the range of recommendations between author- and editor-suggested reviewers was the same, it was interpreted as no preference.

Statistical Analysis

Missing data for individual items of the RQI were imputed by best subset regression from the remaining items using data from both raters. The agreement between raters was assessed using the weighted κ statistic.8 To compare RQI scores, we first calculated the mean of the 2 raters' scores. Where there were 2 or more author-suggested reviews for a study, we calculated the mean and repeated this for studies with 2 or more editor-suggested reviews. The difference in the mean RQI scores between author- and editor-suggested reviewers was assessed using a paired t test.

Differences between author- and editor-suggested reviewers in their recommendations to accept (as opposed to revise or reject) were assessed using odds ratios (ORs) from conditional logistic regression (conditional on the paper) and repeated for a recommendation to accept or revise (as opposed to reject). The data were first analyzed excluding data from BMJ. We then examined whether the effect of reviewer source on recommendation differed between papers submitted to BMJ (in which the identity of reviewers is known to authors) and the other journals (in which authors are blinded to reviewer identity) by using a likelihood ratio test on the interaction between reviewer source and whether the reviewer's identity was revealed.

For papers where author- and editor-suggested reviewers differed in their recommendations, we assessed whether the final journal decision (accept or reject) was more likely to reflect the author- or editor-suggested reviewers' preferences. The reject category included cases in which authors failed to resubmit a revised manuscript.

For all comparisons between author- and editor-suggested reviewers, the unit of analysis was the paper. All statistical analyses were performed using STATA software version 8.2 (Stata Corporation, College Station, Tex).

In 48% (1471/3014) of papers sent out for review, the authors suggested at least 1 reviewer (Table 1). There were 329 manuscripts for which at least 1 author-suggested and 1 editor-suggested reviewer were obtained and there were 788 reviews of these manuscripts. Agreement between raters was moderate (κw, 0.56; 95% confidence interval, 0.49-0.63) but consistent with previous research.5

Review Quality and Recommendation for Publication

Review quality did not differ greatly between author- and editor-suggested reviewers (Table 2). However, author-suggested reviewers were more likely to provide a favorable recommendation (accept and revise) in the 6 journals that solicited recommendations with blinded reviews. The extent to which author-suggested reviewers provided more favorable recommendations for acceptance was even greater for open (BMJ) reviews (test for interaction P = .02).

Table Graphic Jump LocationTable 2. Impact of Reviewer Status on Review Quality and Recommendation to Publish
Reviewer Recommendation and Journal Decision to Publish

There were 106 manuscripts in which author- and editor-suggested reviewers differed in their recommendations to publish, with author-suggested reviewers giving more favorable recommendations in 75 (70.8%) of these reviews (Table 3). However, the final editorial decision to accept or reject a study was evenly balanced with 54 (50.9%) decisions consistent with author-suggested reviewers' preferences (30 with more favorable recommendations accepted, 24 with less favorable recommendations rejected). Decisions about the other 52 (49.1%) studies were consistent with the editor-suggested reviewers' preferences.

Table Graphic Jump LocationTable 3. Association Between Final Journal Decision and Who Provided the More Favorable Review (n = 106)

Author- and editor-suggested reviewers of manuscripts did not differ in the quality of their reviews but author-suggested reviewers tended to make more favorable recommendations for publication, particularly if the reviewers' identity was unblinded to the author. Editors' decisions showed no overall preference between author- and editor-suggested reviewers' recommendations. This is consistent with reviewers of a surgical journal3 and a recent unpublished study with reviewers of BioMed Central's journals.9 Our results are more applicable to other medical journals because we had a larger sample of reviewers from 10 journals in different specialties.

Author-suggested reviewers might make more favorable recommendations because they know the author personally or have received a positive review from the author in the past. However, it is not necessarily the case that authors know their suggested reviewer. A more plausible reason is that authors recommend experts in their field of research who will recognize the importance of their paper. In contrast, while editor-suggested reviewers might work in the authors' specialty, they may be less interested in the issues raised in the paper and even keen to see it rejected.

Our finding that more positive recommendations made by author-suggested reviewers is greater in BMJ (with open reviewers) than among the blinded reviews of other journals should be treated with some caution because there may be other journal characteristics that explain the difference.10,11

Study Limitations

There may have been a Hawthorne effect, ie, while editors were instructed to choose reviewers in their usual manner, they were aware of the objectives of the study and may have altered their behavior. In addition, editorial decisions about manuscripts may have been influenced by the existence of additional reviews from author-suggested reviewers solicited by journal administrators. It is unclear what biases, if any, such factors may have introduced. Only a small proportion of the total number of papers sent for review during the study period were included (Table 1). This was largely due to reviews solicited not resulting in a pair of completed reviews. We conducted an observational study and did not alter the decision-making process.

Study Implications

Our findings suggest that editors can make use of author-suggested reviewers and expect reviews of similar quality, but with the caveat that the recommendation to publish may be more favorable. The latter is not a problem for many journals, including BMJ, because they do not ask reviewers to make a recommendation. The decision to publish is an editorial decision based not only on the scientific review but a number of other factors.

Corresponding Author: Sara Schroter, PhD, BMJ Editorial Office, BMA House, Tavistock Square, London WC1H 9JR, England (sschroter@bmj.com).

Author Contributions: Dr Schroter and Mr Hutchings had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Schroter, Black.

Acquisition of data: Schroter, Tite.

Analysis and interpretation of data: Schroter, Hutchings.

Drafting of the manuscript: Schroter, Hutchings, Black.

Critical revision of the manuscript for important intellectual content: Schroter, Tite, Hutchings.

Statistical analysis: Hutchings.

Administrative, technical, or material support: Schroter, Tite.

Study supervision: Schroter, Black.

Financial Disclosures: None reported.

Funding/Support: This study was funded by the BMJ Publishing Group's research budget.

Previous Presentation: Presented in part at the Fifth International Congress on Peer Review and Biomedical Publication; September 16-18, 2005; Chicago, Ill.

Acknowledgment: We thank Michael Healy, retired statistical advisor emeritus, Archives of Disease in Childhood, and Ben Armstrong, reader in epidemiology statistics, London School of Hygiene & Tropical Medicine, for their unpaid and independent statistical advice.

Evans AT, McNutt RA, Fletcher SW, Fletcher RH. The characteristics of peer reviewers who produce good-quality reviews.  J Gen Intern Med. 1993;8:422-428
PubMed   |  Link to Article
Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review in a general medical journal.  JAMA. 1998;280:231-233
PubMed   |  Link to Article
Earnshaw JJ, Farndon JR, Guillou PJ, Johnson CD, Murie JA, Murray GD. A comparison of reports from referees chosen by authors or journal editors in the peer review process.  Ann R Coll Surg Engl. 2000;82:(4 suppl)  133-135
PubMed
van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts.  J Clin Epidemiol. 1999;52:625-629
PubMed   |  Link to Article
Schroter S, Black N, Evans S, Smith R, Carpenter J, Godlee F. Effects of training on the quality of peer review: a randomised controlled trial.  BMJ. 2004;328:673-675
PubMed   |  Link to Article
van Rooyen S, Godlee F, Smith R, Evans S, Black N. The effect of blinding and unmasking on the quality of peer review: a randomized trial.  JAMA. 1998;280:234-237
PubMed   |  Link to Article
van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial.  BMJ. 1999;318:23-27
PubMed   |  Link to Article
Landis JR, Koch GG. The measurement of observer agreement for categorical data.  Biometrics. 1977;33:159-174
PubMed   |  Link to Article
Wager E, Parkin E, Tamber PS. Are reviewers suggested by authors as good as those chosen by editors? results of a rater-blinded, retrospective study. Abstract published in 5th International Congress on Peer Review and Biomedical Publication conference proceedings; Chicago, Ill; September 2005. Available at: http://www.ama-assn.org/public/peer/prc2005prog.pdf. Accessed December 22, 2005
Rennie D. Editorial peer review: its development and rationale. In: Godlee F, Jefferson T, eds. Peer Review in Health Sciences. London, England: BMJ Books; 1999
Kassirer JP, Campion EW. Peer review: crude and understudied, but indispensable.  JAMA. 1994;272:96-97
PubMed   |  Link to Article

Figures

Tables

Table Graphic Jump LocationTable 1. Manuscripts and Reviewers by Journal
Table Graphic Jump LocationTable 2. Impact of Reviewer Status on Review Quality and Recommendation to Publish
Table Graphic Jump LocationTable 3. Association Between Final Journal Decision and Who Provided the More Favorable Review (n = 106)

References

Evans AT, McNutt RA, Fletcher SW, Fletcher RH. The characteristics of peer reviewers who produce good-quality reviews.  J Gen Intern Med. 1993;8:422-428
PubMed   |  Link to Article
Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review in a general medical journal.  JAMA. 1998;280:231-233
PubMed   |  Link to Article
Earnshaw JJ, Farndon JR, Guillou PJ, Johnson CD, Murie JA, Murray GD. A comparison of reports from referees chosen by authors or journal editors in the peer review process.  Ann R Coll Surg Engl. 2000;82:(4 suppl)  133-135
PubMed
van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts.  J Clin Epidemiol. 1999;52:625-629
PubMed   |  Link to Article
Schroter S, Black N, Evans S, Smith R, Carpenter J, Godlee F. Effects of training on the quality of peer review: a randomised controlled trial.  BMJ. 2004;328:673-675
PubMed   |  Link to Article
van Rooyen S, Godlee F, Smith R, Evans S, Black N. The effect of blinding and unmasking on the quality of peer review: a randomized trial.  JAMA. 1998;280:234-237
PubMed   |  Link to Article
van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial.  BMJ. 1999;318:23-27
PubMed   |  Link to Article
Landis JR, Koch GG. The measurement of observer agreement for categorical data.  Biometrics. 1977;33:159-174
PubMed   |  Link to Article
Wager E, Parkin E, Tamber PS. Are reviewers suggested by authors as good as those chosen by editors? results of a rater-blinded, retrospective study. Abstract published in 5th International Congress on Peer Review and Biomedical Publication conference proceedings; Chicago, Ill; September 2005. Available at: http://www.ama-assn.org/public/peer/prc2005prog.pdf. Accessed December 22, 2005
Rennie D. Editorial peer review: its development and rationale. In: Godlee F, Jefferson T, eds. Peer Review in Health Sciences. London, England: BMJ Books; 1999
Kassirer JP, Campion EW. Peer review: crude and understudied, but indispensable.  JAMA. 1994;272:96-97
PubMed   |  Link to Article

Letters

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 43

Related Content

Customize your page view by dragging & repositioning the boxes below.

Articles Related By Topic
Related Collections