0
Peer Review |

Effects of Editorial Peer Review:  A Systematic Review FREE

Tom Jefferson, MD; Philip Alderson, MBChB; Elizabeth Wager, MA; Frank Davidoff, MD
[+] Author Affiliations

Author Affiliations: Health Reviews Ltd, Rome, Italy (Dr Jefferson); UK Cochrane Centre, Oxford, England (Dr Alderson); Sideview, Princes Risborough, England (Ms Wager); and Annals of Internal Medicine, Philadelphia, Pa (Dr Davidoff).


JAMA. 2002;287(21):2784-2786. doi:10.1001/jama.287.21.2784.
Text Size: A A A
Published online

Context Editorial peer review is widely used to select submissions to journals for publication and is presumed to improve their usefulness. Sufficient research on peer review has been published to consider a synthesis of its effects.

Methods To examine the evidence of the effects of editorial peer-review processes in biomedical journals, we conducted electronic and full-text searches of private and public databases to June 2000 and corresponded with the World Association of Medical Editors, European Association of Science Editors, Council of Science Editors, and researchers in the field to locate comparative studies assessing the effects of any stage of the peer-review process that made some attempt to control for confounding. Nineteen of 135 identified studies fulfilled our criteria. Because of the diversity of study questions, methods, and outcomes, we did not pool results.

Results Nine studies considered the effects of concealing reviewer/author identity. Four studies suggested that concealing reviewer or author identity affected review quality (mostly positively); however, methodological limitations make their findings ambiguous, and other studies' results were either negative or inconclusive. One study suggested that a statistical checklist can improve report quality, but another failed to find an effect of publishing another checklist. One study found no evidence that training referees improves performance and another showed increased interrater reliability; both used open designs, making interpretation difficult. Two studies of how journals communicate with reviewers did not demonstrate any effect on review quality. One study failed to show reviewer bias, but the findings may not be generalizable. One nonrandomized study compared the quality of articles published in peer-reviewed vs other journals. Two studies showed that editorial processes make articles more readable and improve the quality of reporting, but the findings may have limited generalizability to other journals.

Conclusions Editorial peer review, although widely used, is largely untested and its effects are uncertain.

The use of peers to assess the work of fellow scientists goes back at least 200 years.1 It is usually assumed to raise the quality of the end product and to provide a mechanism for rational, fair, and objective decision making. Despite the fact that peer review has such a long history and is so well established, research into its effects is a recent phenomenon. However, the body of original research on the effects of peer review has been growing, and systematic review and synthesis may now be possible.2 This review assesses the effects of processes undertaken as part of editorial peer review of original research studies submitted for paper or electronic publication in biomedical journals.

We used Cochrane methods to carry out our review, which will also be published in the Cochrane Database of Methodology Reviews.3 A detailed description of the search strategy, sources, and terms used is available in online Table A (available in PDF format).46

We considered for inclusion all types of comparative studies in which some attempt to control for confounding had been made. This was an attempt to widen the criteria to collect as much evidence as possible, without collecting studies in which obvious confounding might explain any observed effect. We considered studies assessing the effects of any stage of the peer review process except those solely concerned with technical editing, which was the subject of another review.7 For example, we looked for studies of the effects of different ways of assigning submissions to assessors and eliciting opinions, decision-making procedures, and methods of feedback to authors.

Two reviewers (T.J. and P.A.) examined all retrieved citations. Studies for possible inclusion were retrieved in full. The same 2 reviewers then examined studies independently against the selection criteria. The same reviewers extracted information on study design and outcomes. We collected descriptive information on study quality, as reported herein. We identified 135 reports of studies that could possibly fulfill our inclusion criteria. Nineteen of these fulfilled our criteria and we excluded the remaining 116 studies from our review.

We did not pool the results of similar studies into a formal meta-analysis because no 2 studies were alike and all asked a slightly different study question or used different designs or outcome measures. We decided to group studies by the broad issues they addressed.

A summary of the 19 included studies is shown online Table B (available in PDF format ). A more detailed description will be available in our Cochrane review.3

We included 9 studies assessing the effect of blinding/masking or revealing authors' and/or reviewers' identities in some way (eg, exchanging reviews between peer reviewers) on the quality of external reviews.816 All of the studies in this group were randomized controlled trials. Five studies1115 reported no apparent effect in quality of reviews, time taken to review, or tone of reviews. The 4 studies810,16 that suggested that concealing reviewer or author identity affected review quality had methodological limitations that make their outcomes ambiguous. The most interesting methodological issue we identified in the design of the studies was the consistent difficulty in ensuring robust blinding procedures. This was probably a reflection of the relatively small world of editorial peer review, especially when applied to specialist areas of knowledge. Overall, most of the studies were inconclusive because blinding was ineffective and confidence intervals were not sufficiently narrow to allow firm conclusions to be made.

We included 2 studies that assessed the effects of submission checklists on outcomes.17,18 The study by Gardner and Bond 18 shows some benefit from using a statistical checklist, although this finding may be of limited generalizability given the setting of the study (BMJ) and its small size. The other study found that publication of a checklist for economic studies did not appear to improve submissions.17

Two studies assessed the effects of the media used by journals to communicate with reviewers. One found that posting to the Internet may benefit authors but does not appear to affect the quality of reviews.19 The randomized controlled trial by Neuhauser and Koran20 showed that warning reviewers of an impending review by telephone lengthened total turnaround time, although it shortened review time.

Two studies assessed the effects of training reviewers.21,22 The results were ambiguous in the study by Callaham et al,21 showing no obvious effect, in contrast with the results of Strayhorn et al,22 which showed that interrater reliability increased. The open design of both studies makes minimization of biases impossible, and results are therefore difficult to interpret. Generalizability of results beyond the context of the specific interventions tested is questionable.

A randomized controlled trial by Ernst and Resch23 assessed the presence and effects of reviewer bias. They found no evidence of reviewer bias toward an unconventional treatment. Despite its randomized design, bias may distort the results of this study (the response rate was 61%).

We included 1 study assessing the effects of peer review on study validity,24 which failed to show any difference, although the small size and nonrandomized design of the study again makes generalizability of its results uncertain.

Two studies assessed the effects of peer review on study report quality.25,26 Both studies showed a beneficial effect, but results may again have limited generalizability because of atypical settings (both journals studied are well resourced and keen on improving quality).

In summary, the quality of randomized studies was superior to that of trials with open design. (Quality refers to the ability of a study to answer its research question.) Overall, the small numbers of reviews and reviewers involved, potentially atypical settings (with a few major journals being the object of many studies), and many methodological weaknesses make both the internal and external validity of these studies difficult to interpret.

Our review was limited to evidence about peer review as practiced by biomedical journals. We discovered remarkably few well-designed studies assessing the effects of this process. Of the studies identified, the majority were focused on specific editorial processes, such as masking of reviewers and authors or use of checklists. Very few examined the broader effects of peer review. We assess the implications of this in a separate study.27 The only study that attempted to address broader issues24 is difficult to interpret because of methodological weaknesses and the likelihood of limited generalizability. The results of our systematic review are consistent with those of an earlier descriptive review.2

Given the widespread use of peer review and its importance, it is surprising that so little is known of its effects. However, the research needed to address these questions would require a well-funded and coordinated effort involving several sectors of the scientific community as well as the cooperation of large numbers of authors and editors, and the methodological issues in conducting proper studies of the subject are daunting.

The 2 principal functions of peer review, filtering out incorrect or inadequate work and improving the accuracy and clarity of published reports, directly reflect the 2 virtues that Francis Bacon in 1605 attributed to doubt: guarding "against errors" and causing issues that "would have been passed by lightly without intervention" to be "attentively and carefully observed." As such, the doubts contributed by peer review are an intrinsic and essential part of science, layered over the critical reviews that take place during the process of doing the work. Future research in biomedical peer review needs to recognize these basic concepts and must consider a number of specific underlying methodological issues. Among these are both the positive contributions of peer review and its abuses, the complexity of biomedical science, and the incremental nature of scientific progress. This is based on rare major discoveries and very frequent minor ones, clarifying patterns of life and disease and society's response to them. Moreover, most studies of biomedical journal peer review have been concerned with manuscripts that were accepted for publication. A number of important questions about peer review can only be answered, however, by studying rejected manuscripts as well as those that are accepted.

Until such research is undertaken, peer review should be regarded as an untested process with uncertain outcomes.

Kronick DA. Peer-review in 18th-century scientific journalism.  JAMA.1990;263:1321-1322.
Overbeke J. The state of the evidence: what we know and what we don't know about journal peer review. In: Godlee F, Jefferson T, eds. Peer Review in Health Sciences. London, England: BMJ Books; 1999:32-45.
Alderson P, Davidoff F, Jefferson TO, Wager E. Editorial peer review for improving the quality of reports of biomedical studies [Protocol for a Cochrane Methodology Review]. Oxford, England: Cochrane Library, Update Software; 2001; issue 3.
Rennie D. Editorial peer review in biomedical publication.  JAMA.1990;263:1317.
Rennie D, Flanagin A. The second International Congress on Peer Review in Biomedical Publication.  JAMA.1994;272:91.
Rennie D, Flanagin A. Congress on Biomedical Peer Review.  JAMA.1998;280:213.
Wager E, Middleton P. Effects of technical editing in biomedical journals: a systematic review.  JAMA.2002;287:2821-2824.
McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review.  JAMA.1990;263:1371-1376.
Fisher M, Friedman SB, Strauss B. The effects of blinding on acceptance of research papers by peer review.  JAMA.1994;272:143-146.
Jadad AR, Moore A, Carroll D.  et al.  Assessing the quality of reports of randomized clinical trials.  Control Clin Trials.1996;17:1-12.
van Rooyen S, Godlee F, Evans S.  et al.  Effect of blinding and unmasking on the quality of peer review.  JAMA.1998;280:234-237.
Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding peer reviewers and asking them to sign their reports.  JAMA.1998;280:237-240.
Justice AC, Cho MK, Winker MA.  et al.  Does masking author identity improve peer review quality?  JAMA.1998;280:240-242.
van Rooyen S, Godlee F, Evans S.  et al.  Effect of open peer review on quality of reviews and on reviewers' recommendations.  BMJ.1999;318:23-27.
Das Sinha S, Sahni P, Nundy S. Does exchanging comments of Indian and non-Indian reviewers improve the quality of manuscript reviews?  Natl Med J India.1999;12:210-213.
Walsh E, Rooney M, Appleby L, Wilkinson G. Open peer review.  Br J Psychiatry.2000;176:47-51.
Jefferson T, Smith R, Yee Y.  et al.  Evaluating the BMJ guidelines for economic submissions.  JAMA.1998;280:275-277.
Gardner MJ, Bond J. An exploratory study of statistical assessment of papers published in the British Medical Journal JAMA.1990;263:1355-1357.
Bingham CM, Higgins G, Coleman R, Van der Weyden MB. The Medical Journal of Australia Internet peer-review study.  Lancet.1998;352:441-445.
Neuhauser D, Koran CJ. Calling Medical Care reviewers first.  Med Care.1989;27:664-666.
Callaham ML, Wears RL, Waeckerle JF. Effect of attendance at a training session on peer reviewer quality and performance.  Ann Emerg Med.1998;32:318-322.
Strayhorn J, McDermott Jr JF, Tanguay P. An intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry Am J Psychiatry.1993;150:947-952.
Ernst E, Resch KL. Reviewer bias against the unconventional?  Complement Ther Med.1999;7:19-23.
Elvik R. Are road safety evaluation studies published in peer reviewed journals more valid than similar studies not published in peer reviewed journals?  Accid Anal Prev.1998;30:101-118.
Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine Ann Intern Med.1994;121:11-21.
Pierie JP, Walvoort HC, Overbeke AJ. Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde Lancet.1996;348:1480-1483.
Jefferson T, Wager E, Davidoff F. Measuring the quality of editorial peer review.  JAMA.2002;287:2786-2790.

Figures

Tables

References

Kronick DA. Peer-review in 18th-century scientific journalism.  JAMA.1990;263:1321-1322.
Overbeke J. The state of the evidence: what we know and what we don't know about journal peer review. In: Godlee F, Jefferson T, eds. Peer Review in Health Sciences. London, England: BMJ Books; 1999:32-45.
Alderson P, Davidoff F, Jefferson TO, Wager E. Editorial peer review for improving the quality of reports of biomedical studies [Protocol for a Cochrane Methodology Review]. Oxford, England: Cochrane Library, Update Software; 2001; issue 3.
Rennie D. Editorial peer review in biomedical publication.  JAMA.1990;263:1317.
Rennie D, Flanagin A. The second International Congress on Peer Review in Biomedical Publication.  JAMA.1994;272:91.
Rennie D, Flanagin A. Congress on Biomedical Peer Review.  JAMA.1998;280:213.
Wager E, Middleton P. Effects of technical editing in biomedical journals: a systematic review.  JAMA.2002;287:2821-2824.
McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review.  JAMA.1990;263:1371-1376.
Fisher M, Friedman SB, Strauss B. The effects of blinding on acceptance of research papers by peer review.  JAMA.1994;272:143-146.
Jadad AR, Moore A, Carroll D.  et al.  Assessing the quality of reports of randomized clinical trials.  Control Clin Trials.1996;17:1-12.
van Rooyen S, Godlee F, Evans S.  et al.  Effect of blinding and unmasking on the quality of peer review.  JAMA.1998;280:234-237.
Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding peer reviewers and asking them to sign their reports.  JAMA.1998;280:237-240.
Justice AC, Cho MK, Winker MA.  et al.  Does masking author identity improve peer review quality?  JAMA.1998;280:240-242.
van Rooyen S, Godlee F, Evans S.  et al.  Effect of open peer review on quality of reviews and on reviewers' recommendations.  BMJ.1999;318:23-27.
Das Sinha S, Sahni P, Nundy S. Does exchanging comments of Indian and non-Indian reviewers improve the quality of manuscript reviews?  Natl Med J India.1999;12:210-213.
Walsh E, Rooney M, Appleby L, Wilkinson G. Open peer review.  Br J Psychiatry.2000;176:47-51.
Jefferson T, Smith R, Yee Y.  et al.  Evaluating the BMJ guidelines for economic submissions.  JAMA.1998;280:275-277.
Gardner MJ, Bond J. An exploratory study of statistical assessment of papers published in the British Medical Journal JAMA.1990;263:1355-1357.
Bingham CM, Higgins G, Coleman R, Van der Weyden MB. The Medical Journal of Australia Internet peer-review study.  Lancet.1998;352:441-445.
Neuhauser D, Koran CJ. Calling Medical Care reviewers first.  Med Care.1989;27:664-666.
Callaham ML, Wears RL, Waeckerle JF. Effect of attendance at a training session on peer reviewer quality and performance.  Ann Emerg Med.1998;32:318-322.
Strayhorn J, McDermott Jr JF, Tanguay P. An intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry Am J Psychiatry.1993;150:947-952.
Ernst E, Resch KL. Reviewer bias against the unconventional?  Complement Ther Med.1999;7:19-23.
Elvik R. Are road safety evaluation studies published in peer reviewed journals more valid than similar studies not published in peer reviewed journals?  Accid Anal Prev.1998;30:101-118.
Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine Ann Intern Med.1994;121:11-21.
Pierie JP, Walvoort HC, Overbeke AJ. Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde Lancet.1996;348:1480-1483.
Jefferson T, Wager E, Davidoff F. Measuring the quality of editorial peer review.  JAMA.2002;287:2786-2790.
CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 117

Related Content

Customize your page view by dragging & repositioning the boxes below.