Making Reviewers Visible: Title and subTitle BreakOpenness, Accountability, and Credit

The system of prepublication peer review in biomedical science, which seems as Judson¹ described in THE JOURNAL in 1994 "so monolithic and unchangeable," has existed only since World War II. One feature that is seen by many as an essential part of the monolith is anonymity for peer reviewers. Intermittent calls for reviewers to sign their reviews, either voluntarily or as a requirement,^{2 - 4} have met with little result. While anonymity has disappeared from most walks of life and certainly most areas of publishing, and despite the fact that most journals do not confer anonymity on authors undergoing review, anonymity for peer reviewers is still the overwhelming norm.

There are strong arguments in favor of retaining reviewer anonymity. As summarized in a series of commentaries commissioned by the editor of Cardiovascular Research in 1994,⁵ they include concerns that in an open system junior reviewers would be less likely to give honest criticism of work by senior colleagues, established scientists would be favored by "old boy" networks, reviewers would be less critical, acceptance rates would rise, resentment and animosity between researchers would increase, and editors would be under greater pressure. To counter these concerns, I suggest 4 arguments in favor of open review, all of which (and many more) have been outlined by others.^{2 - 6}

First, open review is ethically superior to anonymous review. Despite raising the practical concerns already listed, most of the commentators in Cardiovascular Research spoke in favor of open review on ethical grounds.⁵ Open review, they argued, would increase the accountability of the reviewer, giving less scope for biased or unjustified judgments or misappropriation of data under the cloak of anonymity. Several of the commentators stressed the inequity of a system in which authors are identified and reviewers are not. Rennie⁷ wrote:

The only ethically justifiable systems of peer review are either completely closed (with no one but an editorial assistant knowing the identity of the authors and only the editor knowing the identity of the reviewer) or completely open.

A "completely closed" system has, to my knowledge, never been attempted, and studies have found that less rigorous systems, which attempt only to conceal authors' identity from reviewers, are unlikely to succeed. In 4 randomized controlled trials, reviewers who were not told the identity of authors correctly identified the authors in 23% to 42% of cases.^{8 - 11} If as one might expect, the identities of better-known authors are harder to conceal, this patchy success of blinding would be likely to introduce a systematic bias, whether in their favor or against them. By Rennie's reckoning, this leaves only one truly ethical alternative: a system in which authors and reviewers know one another's identities.

My second argument in favor of open peer review is that requiring reviewers to sign their reports does not seem to adversely affect the usefulness of their comments. Four of 5 trials of open review have found no difference in review quality as judged by editors and authors^{8 - 9 ,12} (van Rooyen et al, unpublished data, 2002), while one found improvement with open review¹³ (Table 1). This is not to say that open review is without its problems. The trials also suggest that open review increases the number of reviewers who decline to review, the likelihood that reviewers will recommend acceptance, and the time taken to produce a report. Those who remain unconvinced by the ethical arguments and who fear that open review will increase acrimony and back scratching in science will focus on these things. Those who wish to introduce open review on ethical grounds will focus on the lack of important adverse effects. This was the view taken at the BMJ, which introduced open review in 1999,⁶ and at BioMed Central's medical journals, which have adopted a system, evaluated at the BMJ in a randomized controlled trial (van Rooyen et al, unpublished data, 2002), in which signed reviews and authors' responses are posted on the Internet alongside accepted articles.

Those of us adopting open review may seem to be rushing ahead on limited evidence. Three of the 5 trials were performed at 1 journal, and 4 of the 5 used the same instrument for assessing quality, which, although properly validated,¹⁴ may miss important aspects of review quality. On the other hand, as Rennie argued in 1998, it is not open review that should have to justify itself but the "ethically unequal and inconsistent system" practiced by most journals, in which authors are identified and peer reviewers are not.⁴ The available evidence gives no indication that anonymous peer review achieves better scientific results than open review.

My third argument is that open review is feasible in practice. Experience at the BMJ and BioMed Central shows that open review works. Authors like it; some reviewers decline to review openly but others welcome it; authors can identify conflicts of interest that reviewers have failed to declare and editors are not in a position to detect; no adverse effects (such as careers ruined or reviewers beaten up) have been reported via the BMJ's yellow card alerting system; and no reviewers have made unacceptable comments. Indeed, as expected, signed reviews seem more constructive in their tone; and while it may sometimes be harder to find reviewers, any increase in editorial work is balanced by the time saved in not having to edit reviewers' comments. If more reviewers recommend acceptance, this should not adversely affect either the journal or science, since most journals do not rely solely on reviewers to decide which articles to publish. In addition, posting signed reviews on the Internet alongside published articles puts reviewers squarely in front of their own peers and should make them less likely to give a bad article an easy ride.

My fourth argument is that open review can provide reviewers with credit for the work they do, which should balance the demands associated with greater accountability. The current system confers little credit on reviewers, either from their peers or from their academic institutions. Reviewing can seem a thankless task, performed in isolation with little or no feedback and no obvious reward. Journals rarely pay their reviewers: most use a range of other rewards, including naming reviewers in the print journal or on the Internet, writing a letter of thanks, and writing letters of reference for their best reviewers to show to promotion and tenure committees. Posting signed reviews on the Internet and making them accessible through common search engines would achieve wider recognition for peer review within the scientific community. Web sites could highlight the best reviews, as judged by editors and authors, using existing validated tools and new ones yet to be developed. In this way, the quality and quantity of contributions to the peer-review process could be assessed alongside publication record as an additional measure of a researcher's impact in his or her field. Such public credit would create a much more powerful motivation to do a good job than letters of thanks or listing of names. It would also allow others to learn from the best reviewers in ways that are not possible at the moment.

Journals and the research communities they serve are naturally conservative. It is not surprising that they prefer a system they know to one that offers challenges as well as benefits. One of the challenges, which may explain why so few journals have opted for open review, is that open review exposes editors as much as reviewers. In an open system, editors can no longer hide behind the spuriously heightened authority of anonymous reviewers. They have to take full responsibility for their choice of reviewer, their interpretation of the reviewers' comments, and the journal's decision on publication.

Exhortations to adopt open peer review will probably then continue to be futile. But other forces, which promise even greater openness, are already at work.^{15 - 17} Preprint archives and some journals are inviting open commentary on articles posted or submitted for publication.¹⁸ Prominent examples include the high-energy physics archive at Cornell University and the journals Atmospheric Chemistry and Physics and Electronic Transactions on Artificial Intelligence. However, in all of these cases, open commentary is currently coupled with traditional anonymous peer review before work can be formally published in a journal. The Medical Journal of Australia is experimenting with a system of open peer review (in the form of an online discussion between authors, editors, and assigned reviewers) followed by posting of accepted articles on the Internet for open commentary. If adopted by the journal, this would allow authors to revise their articles in line with peer reviewers' comments and then in response to comments from a wider group of scientists before the article is published. The Lancet, BMJ, and BioMed Central all now offer authors the option of posting their work on a preprint server while it undergoes peer review. The BMJ and BioMed Central are also planning real-time online open review of submitted articles, followed by open commentary prior to publication. The possible benefits of such a system include those outlined above for open review, as well as a greater likelihood of important errors being detected.

A fully fledged system of posting prepublication work and open commentary will develop within biomedical research only when journals actively encourage authors to post their articles on the Internet as preprints as opposed to discouraging or prohibiting this as many currently do. Journals will also have to make it worthwhile and easy for experts and others to comment on articles and on other people's comments, which may best be achieved by tying the open commentaries and peer-review reports to systems that accumulate academic credits. The change will involve placing the onus on authors to correct and update their work; linking original articles to subsequent comments, corrections, and revised versions; and improving systems for highlighting the best and most interesting articles for specific audiences so that individuals are not swamped by irrelevant and poor-quality material.

Somewhere along the line, we will have to accept that the point at which an article is "published in its final form" will become blurred. Eventually this will mean abandoning our current attempts to provide systematic prepublication peer review. While many people fear this brave new world, we cannot pretend that the current system is beyond improvement.¹⁹ If replaced by a system of open commentary and ongoing revision, in which responsibility for quality control is shared by many rather than depending on the necessarily subjective judgments of a chosen few, this should not spell disaster for science and offers important benefits. Necessary though prepublication peer review is for print journals, with their space constraints requiring selection for publication and where work once published is hard to correct, it is far less important in the electronic medium, with no space constraints and where it is easy to correct mistakes and link to critical comments and revisions.

These changes will happen sooner than most of us can now imagine. They have the potential to bring better, fairer systems of quality control and dissemination. Perhaps most importantly of all, abandoning formal prepublication peer review would remove false reassurances about the reliability of what is published²⁰ and will allow the scientific community to direct its increasingly stretched resources toward preventing poor-quality research rather than patching it up for publication.