0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Peer Review Congress |

Masking Author Identity in Peer Review:  What Factors Influence Masking Success? FREE

Mildred K. Cho, PhD; Amy C. Justice, MD, PhD; Margaret A. Winker, MD; Jesse A. Berlin, ScD; Joseph F. Waeckerle, MD; Michael L. Callaham, MD; Drummond Rennie, MD; and the PEER Investigators
[+] Author Affiliations

From the Center for Bioethics, University of Pennsylvania, Philadelphia (Dr Cho); Division of General Internal Medicine and Program for Health Care Research, Department of Veterans Affairs Medical Center, Cleveland, Ohio, and University Hospitals of Cleveland, Case Western Reserve University (Dr Justice); Division of Biostatistics, Department of Biostatistics and Epidemiology and Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania School of Medicine, Philadelphia (Dr Berlin); Department of Emergency Medicine, University of Missouri Kansas City School of Medicine (Dr Waeckerle); Division of Emergency Medicine, University of California, San Francisco School of Medicine (Dr Callaham); Institute for Health Policy Studies, University of California, San Francisco (Dr Rennie). Dr Winker is Senior Editor and Dr Rennie is Deputy Editor (West), JAMA.


JAMA. 1998;280(3):243-245. doi:10.1001/jama.280.3.243.
Text Size: A A A
Published online

Context.— In a previous study, we found that masking success was higher at a journal that masked reviewers to author identity. We hypothesized that masking policy or other factors could be associated with masking success.

Objectives.— To evaluate differences in success of masking reviewers to author identity at 7 biomedical journals and to identify factors associated with these differences.

Design.— Written questionnaire.

Participants.— Reviewers at 3 journals with a long-standing policy of masking author identity (Annals of Emergency Medicine, Epidemiology, and Journal of the American Geriatrics Society) and 4 journals without a policy of masking author identity (Annals of Internal Medicine, JAMA , Obstetrics & Gynecology , and Ophthalmology).

Main Outcome Measures.— Masking success (percentage of reviewers successfully masked) and reviewer characteristics associated with masking.

Results.— There was no significant difference in masking success between journals with a policy of masking (60%) and those without (58%) (P=.92). We found no association between masking success and a policy of masking when adjusted for the reviewer characteristics of age, sex, years of reviewing experience, number of articles published, number of articles reviewed, percentage of time spent in research, editorial experience, or academic rank (odds ratio [OR], 1.3; 95% confidence interval [CI], 0.64-2.8; P=.43). In multivariable analysis of reviewer characteristics, reviewers spending a greater percentage of time in research, the only significant predictor of masking success, were less likely to be successfully masked (OR, 1.01; 95% CI, 1.00-1.02) (P=.04).

Conclusions.— Masking success appears unrelated to a journal policy of masking, but is associated with reviewers' research experience and could be affected by other characteristics. Using reviewers with less research and reviewing experience might increase masking success, but the effect on review quality is unknown.

MASKING peer reviewers to author identity has been suggested as a means to improve quality and fairness of reviews. In a previous study,1 we found that masking reviewers to author identity in the peer review process was not uniformly successful at different journals, and that incomplete masking success limited our ability to assess the effect of masking on review quality. In particular, we noted that the Annals of Emergency Medicine had significantly higher rates of masking success than the other journals. Because the Annals of Emergency Medicine was the only journal in our study that had a long-standing policy of masking, we hypothesized that their policy might be responsible for the high rate, either because reviewers were accustomed to not knowing author identity or because the way in which the manuscript was prepared differed from other journals. Differences in characteristics of specialties might also be responsible.

In this study, we explored 2 questions: (1) is masking success associated with having a long-standing policy of masking? and (2) is masking success associated with reviewer characteristics?

Journals and Procedure

In this study we included 7 journals, 3 with a long-standing policy of masking author identity (Annals of Emergency Medicine, Epidemiology, and Journal of the American Geriatrics Society) and 4 journals that did not mask author identity (Annals of Internal Medicine, JAMA , Obstetrics & Gynecology, and Ophthalmology).

Between August and December 1996, each journal enrolled manuscripts that met the following inclusion criteria: (1) the manuscript was sent for external peer review, (2) the manuscript reported original research, including meta-analyses but excluding case reports or letters, and (3) the authors did not object to having their manuscripts enrolled. Each journal masked eligible manuscripts by removing author and institutional identity from the title page, running headers or footers, and acknowledgments of manuscripts. Self-references in the text were not removed. Each journal then sent the manuscript to at least 2 reviewers along with a 1-page questionnaire. The reviewers were asked to return the manuscript and review to the journal and to return the questionnaire to the study coordinator in a separate, preaddressed envelope.

Each journal enrolled at least 20 manuscripts. Approximately one quarter of these were randomized to the usual reviewing process and were not analyzed in this study if the usual process was not to mask reviewers. Data from Annals of Internal Medicine, JAMA , Obstetrics & Gynecology, and Ophthalmology and data from 22 of 42 total manuscripts at Annals of Emergency Medicine were collected in a previous study,1 in which 2 reviewers per manuscript were enrolled, one of whom was masked to author identity. For this study, we used data only from the masked reviewer. For Epidemiology and Journal of the American Geriatrics Society and additional manuscripts from Annals of Emergency Medicine, between 1 and 4 reviewers per manuscript were included in the study, all of whom were masked to author identity.

Reviewer Questionnaire. Using the reviewer questionnaire, we collected data on masking success and reviewer familiarity with the author from all 7 journals and collected data on reviewer characteristics from 6 journals (not Journal of the American Geriatrics Society). (Copies of the questionnaire will be provided to interested readers on request.)

We determined masking success by asking reviewers the question, "Do you think you can identify any of the principal authors of this manuscript?" If they answered yes, we instructed reviewers to list any authors they thought they could identify. If any reviewer correctly identified at least 1 author, we considered the reviewer unmasked to author identity. Thus, we measured masking success based on the percentage of reviewers who were able to guess author identity. For reviewers who thought they knew at least 1 of the authors, we asked whether they were familiar with the work described in the manuscript before receiving it to review and whether the reviewer was familiar with the author's work.

Reviewers were asked to provide information on age, sex, academic rank, reviewing experience, editorial experience, number of original research articles published in the last 5 years (including meta-analyses), number of manuscripts reviewed in the last 3 months, and percentage of time spent in research, clinical, and administrative activities.

Analysis. We used χ2 tests for analysis of categorical variables, and t tests for analysis of continuous variables. We used unconditional logistic regression models with single variables to estimate the unadjusted odds ratio (OR) for the outcome variable—masking success—associated with each of the potential predictor variables. Subsequently, we fit a multivariable logistic regression model including all the potential predictors to estimate the ORs for each variable, adjusted for all other variables in the model. Non–normally distributed continuous variables were also analyzed after log transformation, but transformation did not substantially change the results or conclusions. Results of untransformed data are reported here. For all tests of significance, we used 2-tailed α=.05.

Reviewer Response Rate and Masking Success Rate

The overall response rate of reviewers was 87% (287/328). No eligible reviewers declined to participate in the study. Overall, 60% of reviewers were successfully masked (95% confidence interval [CI], 54%-65%). The response rates and masking success rates by journal are shown in Table 1.

Table Graphic Jump LocationTable 1.—Success of Masking Reviewers by Masking Policy and by Journal

Masking success was significantly higher at Annals of Emergency Medicine (83% [95% CI, 74%-92%]; N=78) than at all other journals combined (N=209) (P<.001). There was no significant difference in masking success between journals with a policy of masking and those without (P=.92). There was also no significant association between masking success and a policy of masking when adjusted for the reviewer characteristics of age, sex, years of reviewing experience, number of articles published, number of articles reviewed, percentage of time spent in research, editorial experience, or academic rank (full professor vs all other ranks) (OR=1.3; 95% CI, 0.64-2.8; P=.43).

Reviewer Characteristics and Factors Associated With Masking Success. We performed univariate analyses of reviewer characteristics that we hypothesized might be related to masking success. For reviewers from all journals combined (except Journal of the American Geriatrics Society, which did not collect data on reviewer characteristics), masking success was not associated with age, academic rank, or having editorial experience (P>.05). However, compared with reviewers who guessed author identity correctly, successfully masked reviewers had fewer years of reviewing experience, published fewer research articles, and spent less time in research (Table 2, unadjusted ORs).

Table Graphic Jump LocationTable 2.—Predictors of Masking Success: Multivariable Analyses of Reviewer Characteristics (at All Journals Except Journal of the American Geriatrics Society)*

In a multivariable logistic regression model including reviewer age, sex, years of reviewing experience, number of articles published, number of articles reviewed, percentage of time spent in research, editorial experience, academic rank (full professor vs all other ranks), and journal (Annals of Emergency Medicine vs all other journals), we investigated the factors associated with correctly identifying an author. The unadjusted OR for masking success for reviewers at Annals of Emergency Medicine, compared with reviewers for other journals, was 4.8 (95% CI, 2.5-9.3; P <.001). However, after adjusting for the other 8 variables, the OR was 3.3 (95% CI, 1.4-7.2; P=.009), indicating a reduced, but still substantial, difference in masking success between Annals of Emergency Medicine and other journals. Thus, reviewer characteristics did not explain most of the difference in ability to identify authors between reviewers at Annals of Emergency Medicine and reviewers at other journals.

Because factors such as age, number of publications, and reviewing experience are likely to be correlated, we performed regression analysis to examine which, if any, of the reviewer characteristics (other than the journal for which they were reviewing or the journal policy of masking) was an independent predictor of masking success. Of the reviewer characteristics we analyzed, percentage of time spent in research was the only significant independent predictor (P=.04; Table 2, adjusted ORs).

Because some manuscripts were reviewed by more than 1 reviewer, we repeated the logistic regression analyses, adjusted for clustering by manuscripts. The results were essentially the same as the unadjusted results (data not shown).

We confirmed the results of our previous study1 that the rate of masking success varies widely across journals, but found that a long-standing policy of masking did not increase masking success. Previous studies25 have found rates of masking success similar to those in this study. In our study, masking success was not associated with having a long-standing policy of masking, per se. In multivariable analysis, we found that masking success was associated with 1 reviewer characteristic. Reviewers spending a greater proportion of time in research were more likely to identify authors, suggesting that using reviewers with less research experience will increase masking success. Because research experience might be a desirable trait in reviewers for many manuscripts, however, excluding reviewers with this trait could be detrimental to the quality of reviewing. Thus, journals that mask author identity may have to balance their desire for improving the success of masking with the need for high-quality reviewers.

Characteristics of reviewers and other factors we examined did not completely explain differences in masking success. One possible factor might have been in differences in masking procedures among journals. Although the minimum masking procedure followed a standard protocol, Annals of Internal Medicine and Annals of Emergency Medicine used additional procedures that may have affected masking success. Annals of Internal Medicine removed names and journal identification (but not titles or other reference information) from self-references in the text and reference section. However, the masking success rate at Annals of Internal Medicine was very close to the overall average. At Annals of Emergency Medicine, the instructions to authors stated that the journal's policy was to mask author identity and requested that authors not include identifiers in running heads, but running heads would have been removed as part of normal study procedures. Although all 3 of the journals in our study with a policy of masking have had these policies for years, different journal "cultures" might have arisen that affected masking success (eg, authors at some journals might be less likely to self-reference).

Characteristics such as the age and size of medical specialties could influence the type of research performed and the probability that researchers will be able to identify others' work. Contrary to what one might expect, however, JAMA and Annals of Internal Medicine, journals representing large and broad fields, had lower masking success rates than Annals of Emergency Medicine and Obstetrics & Gynecology. Our analyses suggest that to determine whether masking might be more successful in one field than another specialty characteristics other than size of the field would need to be explored.

Our results suggest that reviewers' research experience is associated with masking success, but that other unidentified factors are also involved. Indeed, results from a previous study1 suggested that author renown may in part affect masking success; this factor may be related to reviewer experience. In this study, we did not collect data on this author renown for all journals, and were thus unable to further investigate its effects. The study was also limited by the relatively small number of manuscripts enrolled at journals without a policy of masking. Additional research on a larger number of journals from a variety of medical specialties would be needed to determine what other characteristics of reviewers (eg, specialty or their relationships to authors), authors, journals, or medical specialties are associated with masking success.

Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D.and the PEER Investigators.  Does masking author identity improve peer review quality? a randomized controlled trial.  JAMA.1998;280:240-242.
Yankauer A. How blind is blind review?  Am J Public Health.1991;81:843-845.
McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review.  JAMA.1990;263:1371-1376.
Moossy J, Moossy YR. Anonymous authors, anonymous referees: an editorial exploration.  J Neuropath Exp Neurol.1985;44:225-228.
Fisher M, Friedman SB, Strauss B. The effects of blinding on acceptance of research papers by peer review.  JAMA.1994;272:143-146.

Figures

Tables

Table Graphic Jump LocationTable 1.—Success of Masking Reviewers by Masking Policy and by Journal
Table Graphic Jump LocationTable 2.—Predictors of Masking Success: Multivariable Analyses of Reviewer Characteristics (at All Journals Except Journal of the American Geriatrics Society)*

References

Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D.and the PEER Investigators.  Does masking author identity improve peer review quality? a randomized controlled trial.  JAMA.1998;280:240-242.
Yankauer A. How blind is blind review?  Am J Public Health.1991;81:843-845.
McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review.  JAMA.1990;263:1371-1376.
Moossy J, Moossy YR. Anonymous authors, anonymous referees: an editorial exploration.  J Neuropath Exp Neurol.1985;44:225-228.
Fisher M, Friedman SB, Strauss B. The effects of blinding on acceptance of research papers by peer review.  JAMA.1994;272:143-146.

Letters

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 49

Related Content

Customize your page view by dragging & repositioning the boxes below.