Does Masking Author Identity Improve Peer Review Quality?: Title and subTitle BreakA Randomized Controlled Trial FREE

IT HAS BEEN suggested that masking reviewers to author identity would improve the fairness and the quality of peer review¹ because well-known authors' work may be reviewed less critically. Yet only a small fraction of journals routinely mask reviewers.^{2 - 3} When editors are asked why they do not mask, they cite an "overwhelming burden" associated with masking.^{2 - 3} Some question whether it is possible to mask successfully.^{2 ,4}

One study, conducted at a single journal,⁵ demonstrated that the quality of masked reviews was statistically higher than that of unmasked reviews, although that difference was small. We tested the hypothesis that masking peer reviewers to author identity improves the quality of peer review at 5 biomedical journals. To increase the generalizability of our study, we used a masking procedure that is commonly practiced.

Five journals participated in the study: Annals of Emergency Medicine, Annals of Internal Medicine , JAMA , Obstetrics & Gynecology , and Ophthalmology. Only 1 of these journals, Annals of Emergency Medicine, routinely masks reviewers to author identity.

Manuscript Enrollment. Eligible manuscripts were submitted between November 1995 and March 1996 and met the following inclusion criteria: (1) the manuscript reported original research, including meta-analyses but excluding case reports or letters, (2) the manuscript was sent for external peer review, and (3) the authors did not object to having their manuscripts enrolled. Authors were notified that their manuscripts would be included in a study of the peer review process unless they declined, and that declining to have their manuscripts enrolled would not affect any editorial decisions regarding their manuscript. No authors objected.

Masking Procedure. Each journal followed a standardized masking procedure that involved removing author and institutional identity from the title page, running headers or footers, and acknowledgments of the manuscripts. Self-references in the text were not removed. In addition, the managing editor at Annals of Internal Medicine removed names and journal identification (but not titles or other reference information) from self-references in the text and reference section. Annals of Emergency Medicine also stated in their "Information for Authors" that authors should not include author names in the running heads.

Study Design. At each journal, the editor followed the journal's usual procedure in selecting manuscripts for review and identified 2 reviewers for each manuscript. Once reviewers had agreed to review the manuscript, 2 randomizations were performed. The first was weighted to assign 25% of manuscripts to usual practice. These manuscripts were reviewed according to the journal's usual practice (ie, all but those at Annals of Emergency Medicine were unmasked reviews). The manuscripts randomized to the intervention arm had 1 of 2 reviewers randomly selected to receive a manuscript from which the author identity and institution had been removed. The other reviewer received the manuscript with author and institution identified. Randomization was performed using random number tables. In all cases, both reviewers were sent a questionnaire along with a statement that the manuscript was part of a study of peer review (described herein). The reviewers returned the manuscripts, reviews, and questionnaire to the journal. Before sending the reviews to editors and authors, the managing editors at each journal removed from the reviews any information that would reveal whether the manuscript had been masked. The manuscript editor and the corresponding author rated the quality of each review, unaware of the group to which the review had been assigned.

Questionnaires. Questionnaires were completed by editors, authors, and reviewers. Because of a miscommunication with the managing editor, no authors at the Annals of Emergency Medicine were sent surveys. Editor and author surveys each included 4 specific ratings of review quality: how well the review addressed the clinical or research importance of the study, how well it identified the study's strengths and weaknesses, whether reviewers were courteous, and whether reviewers supplied evidence to support their statements. Authors and editors responded to these questions using a 5-point Likert scale for which a score of 5 represented the best quality. Authors and editors also were asked to provide an overall rating of review quality on a 5-point Likert scale. Editors were asked how much the review had influenced their decision. These questions were chosen to parallel those used in the prior study of masking.⁵

To determine masking success, reviewers in the masked group were asked whether they thought they could identify any of the authors or their institutions, and if so, to list the authors and institutions. Reviewers in both groups were asked whether they were familiar with the authors, their previous work, or the reviewed work.

Analysis. The primary outcome was the difference in quality as assessed by the editor and author between the masked and unmasked review for each manuscript. Because this analysis uses a comparison between the quality of a masked and an unmasked review for each manuscript, a paired t test was used. The sample size for this analysis was the total number of manuscripts randomized to the intervention for which masking status and editor's quality score were complete. A positive difference means that masked reviews were of better quality than unmasked reviews; a negative difference means that masked reviews were of lesser quality than unmasked reviews. The Wilcoxon matched-pairs signed rank test was used to verify the result of the paired t test. A difference of greater than 0.5 on the 5-point Likert scale was considered editorially important.

The secondary outcome was masking success. We defined a review as successfully masked if the reviewer did not guess the author's identity or if the reviewer guessed incorrectly. Exact 95% confidence intervals (CIs) were calculated using the binomial distribution. This analysis did not require a pair of reviews per manuscript. Thus, the sample size for this analysis was all reviews that were randomized to be masked, either because of the usual practice at that journal or because that reviewer was randomized to receive the masking intervention. We analyzed masking success overall and by journal and tested for differences across the 5 journals using the χ²₄ test. We subsequently identified 1 journal with a significantly higher success rate comparing each journal with all other journals using a χ²₁ test. Author renown was determined by whether the randomly unmasked reviewer was familiar with the author, their prior work, or the current work. Logistic regression, with variance estimates adjusted for multiple observations per manuscript, was used to explore whether the higher success of masking at the Annals of Emergency Medicine could be explained by differences in author renown and to test whether author renown was associated with reduced masking success.

A total of 118 manuscripts were randomized, 26 to usual practice and 92 to the intervention. Of those randomized to the intervention, 77 (84%) of 92 had sufficient data to compare review quality based on the editor's judgment. Only 40 (54%) of 74 had sufficient data to compare review quality based on the author's judgment (the denominator for author judgment is smaller because no authors received surveys from Annals of Emergency Medicine). Of those reviewers randomized to receive masked manuscripts, 99 (93%) of 106 had sufficiently complete data to evaluate the success of masking reviewers to author identity.

Review Quality. Editors perceived no significant difference in quality between masked and unmasked reviews (mean difference, 0.1; 95% CI,−0.2 to 0.4). Results were similar when editors rated the degree to which the review influenced their decision (mean difference, –0.1; 95% CI,−0.3 to 0.3). Authors also perceived no overall difference in quality between masked and unmasked reviews (mean difference,−0.1; 95% CI,−0.5 to 0.4). Differences in quality did not vary substantially by journal (Table 1). When the analysis of review quality was restricted to pairs for which masking was successful, no difference in quality was found. There were also no significant quality differences between masked and unmasked reviews for the 4 specific components of quality (data not shown). All results were confirmed when tested using the nonparametric signed rank test.

Masking Success. Success in masking reviewers to author identity was generally low (68%; 95% CI, 58%-77%) and fairly consistent across all but 1 participating journal, Annals of Emergency Medicine, which achieved a masking success rate of 90% (95% CI, 73%-98%; Table 2). This corresponds to odds of success 6.5 times that of the other journals (95% CI, 1.8-23.7). When this journal was excluded from the calculation of overall masking success, the rate dropped to 58% (95% CI, 45%-70%) and a χ² test for significant differences among the remaining journals was not significant.

Author Renown. Manuscripts by authors with whom the unmasked reviewer was familiar (n=43) were less likely to be successfully masked (53%) (that is, the masked reviewer was more likely to correctly guess author identity) than those of authors who were not known to the unmasked reviewer (79%; P =.008; odds ratio [OR], 0.3; 95% CI, 0.1-0.8).

We also considered whether author renown explained the apparent discrepancy in masking success rates between the Annals of Emergency Medicine and the other participating journals. When an indicator variable for Annals of Emergency Medicine was used to predict masking success in a logistic model, the unadjusted OR was 5.9 (95% CI, 1.6-21.1). When a multivariate logistic model was used to adjust for author renown, the OR was reduced but remained significant (OR, 4.8; 95% CI, 1.2-19.3).

McNutt and colleagues⁵ found that reviews of masked manuscripts were of marginally higher quality than reviews of unmasked manuscripts (3.5 vs 3.1 on a 5-point scale). However, they recognized that this difference was small. We believed it was too small to justify the added time and cost of masking. We sought to determine whether the difference in quality due to masking, studied among several journals, might be larger. In this first multijournal study of peer review at biomedical journals, our 95% CI excluded an overall improvement in peer review quality of 0.5 or greater on a 5-point Likert scale, the difference we considered editorially significant.

Poor overall masking success, in combination with the observation that an author's renown is strongly associated with masking failure, is a possible explanation for this finding. Notably, the average masking success in our study was similar to that achieved by McNutt et al⁵ and Yankauer⁶ and was obtained using commonly used procedures that are generalizable to standard journal practices.

The participation of multiple journals, sufficient overall sample size, and a feasible method of masking author identity make it likely that these findings are valid for most biomedical journals. However, our study was limited to biomedical journals and our sample size was too small to eliminate the possibility of an editorially important difference in quality for individual journals. Although we achieved a good response rate for editor's quality ratings (84%), the rate for authors was low (54%). Thus, our conclusions for author evaluations may not generalize well. Additionally, if the major improvement in review quality provided by masking would be expected to occur for manuscripts of renowned authors, then we cannot exclude the possibility that increasing the rate of successfully masking renowned authors could improve review quality. Finally, reviewers were aware that they were participating in a study. The effect of such knowledge on the quality of review is unknown.

Our study did not directly address the question of whether masking improves fairness. However, if masking is frequently unsuccessful, especially among well-known authors, it is not likely to improve fairness, no matter how fairness might be defined. The only potential benefit to a policy of masking that is largely unsuccessful is the appearance of fairness.

We conclude that masking reviewers to author identity as commonly practiced does not improve review quality. Further, masking as commonly practiced fails to hide the identity of renowned authors and therefore may also fail to improve the fairness of review. Techniques to improve masking success are needed to determine whether masking the identity of renowned authors improves review quality or fairness.