Reliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts FREE

LITTLE IS KNOWN about assessing the quality of peer review. Most journals do not have standardized methods of selecting reviewers, nor do they screen or train them. We conducted a study to determine if a subjective rating of review quality is a reliable measure; whether it can be easily replaced by more objective reviewer performance statistics, such as individual rate of recommendation for acceptance; and whether ratings are correlated with reviewer accuracy in detecting errors in manuscripts.

When making decisions about manuscripts, editors at Annals of Emergency Medicine routinely rate the quality of each peer review on a subjective ordinal scale developed by senior editors. The scale ranges from 1 (poor) to 5 (excellent) and is not specifically defined other than as "review quality." No special training in use of the scale is provided, although all editors receive a detailed job description plus the reviewer orientation described below.

All manuscript peer reviews that had been rated by any editors between January 1, 1994, and June 1, 1997, were eligible for study. When recruited by the editorial board, reviewers receive a 4800-word, written orientation on journal expectations for reviews and, with each subsequent review, fill out a form rating 12 specific components of study design, originality, interest for readers, and manuscript quality, plus free text comments for editors and authors. Reviewers are routinely asked to recommend whether each manuscript should be accepted.

Routine performance statistics are calculated by the journal for each reviewer. They include the rate of recommending acceptance and rejection and decision congruence, defined as whether the reviewer recommendation for each manuscript matched the editor's final decision regarding manuscript acceptance.

The quality ratings were examined for interreviewer agreement using intraclass correlation coefficients (ICCs) because reviews received only 1 editorial rating each, and the ICC takes into account the varying number of scores per reviewer while still weighting each "data point" (paper, editor, and the like) only once.

As part of a separate study, in fall 1994 all active reviewers were separately sent a fictitious manuscript, which contained 23 deliberate flaws and was masked to its true purpose, to review. The method is described in detail elsewhere.^{1 - 2} The performance of reviewers who reviewed the manuscript (and had at least 3 rated reviews) in reporting manuscript flaws was compared with the ratings above.

During the study, 4644 reviews (2922 [63%] of them rated) were performed by 756 reviewers on 1551 manuscripts. To focus on regular (not occasional guest) reviewers, reviewers with 2 or less reviews were excluded, leaving 4161 reviews by 395 reviewers. A total of 2686 (65%) of these were rated by 36 editors, involving 973 manuscripts. The 395 peer reviewers who were studied each reviewed an average of 10.5 manuscripts (95% confidence interval [CI], 9.8-11.2), 6.8 of which were rated (95% CI, 6.3-7.3), receiving an average score of 3.7 (95% CI, 3.58-3.73). They recommended acceptance in 26% of their reviews, rejection in 34%, and revision in 40%. Their decisions were congruent with editors in 50% of these reviews.

Thirty-five percent of reviews were not rated by editors, presumably because of workload, and were therefore excluded. These did not differ in rate of recommendation for acceptance, decision congruence rate, or other identifiable characteristics from rated reviews.

With higher review ratings, the editor's decision was more likely to match the reviewer's recommendation. Reviews rated 1 (poor) demonstrated a 21% congruence between editor and reviewer; those rated a 2, 27% congruence; 3, 41% congruence; 4, 55% congruence; and 5 (excellent), 62% congruence. The average rating of reviews with no decision congruence was 3.51 (SE, 0.03) vs 3.99 (SE, 0.03) for those with congruence (P<.001). Review ratings were not associated with the specific decision made by the editor, however. Accepted papers had lower review scores (3.48; SE, 0.13) than papers sent back for revision (3.74; SE, 0.03) or rejected (3.78; SE, 0.03) (P=.02). The weighted κ between reviewers and editors for decision congruence was 0.034.

The ICC was 0.44 for reviewers, 0.24 for editors, and 0.12 for manuscripts, meaning that about 20% of the variance seen in the review ratings was attributable to the reviewer, 6% to the editor, and 1% to the manuscript.

The reviewer's average quality rating correlated poorly with routine performance statistics. The Spearman rank correlation coefficient was −0.34 for the rate of recommendation for acceptance, 0.12 for the rejection rate, and 0.26 for the decision congruence rate.

Seventy-eight percent of all reviewers available to review in fall 1994 (a typical proportion) evaluated the fictitious manuscript. Sufficient ratings were available for 124 reviewers, who also reviewed 2143 genuine manuscripts during the study period and were rated by 30 editors, receiving a mean rating of 3.65 (SD, 0.7). These reviewers had an average of 12 (SD, 5.3) years of experience since residency; 40% were assistant professors, 38% were associate professors, and 17% were full professors.

The reviewers detected a mean 3.4 (SD, 1.6) of the 10 major flaws (those that invalidated or undermined the methods of the study) and 3.1 (SD, 1.7) of the 13 minor flaws (those not affecting study validity) in the fictitious manuscript. The mean editorial rating for each reviewer was modestly correlated with the number of flaws they detected (R=0.45 for major errors, 0.45 for minor errors, and 0.53 for total errors). Reviewers with average ratings of 4 or more reported about twice as many total errors as those with average ratings of 3 or less. Individual reviewer rate of recommendation for acceptance, decision congruence, and various measures of experience were not associated with detection of errors, nor was academic rank (Table 1).

Half of journal editors rely almost exclusively on reviewer recommendations when making acceptance decisions.³ Maintaining the quality of peer reviewers is therefore essential, but most journals have too many reviewers for editors to know their capabilities personally. Most journals do not train reviewers or assess their background in research methodology or critical literature review.⁴ Editors need a simple rating system to monitor review quality, but this rating should require minimal additional work and should add unique value.

Although the reliability of ratings of manuscripts by reviewers has been previously examined, there is only 1 prior study that examined the reliability of ratings of reviewers by editors.⁵ (An earlier study used a 9-axis scale of review quality, but did not test the scale's reliability.⁶ ) The study by Feurer et al⁵ examined 53 reviews of 23 manuscripts, rated by 3 editors. Editors used a scale of 7 axes with scores of varying complexity, guided by a detailed written explanation. The scale included timeliness standards, whether the reviewer wrote on the manuscript or provided supporting references, and other criteria not directly related to quality of scientific assessment. It demonstrated an excellent ICC of 0.84.

By contrast, we chose a scale that was simpler to use and provided a fair to good ICC of 0.44. It performed in a satisfactory fashion compared with scales used by a specially trained panel of experienced researchers in rating manuscripts^{7 - 8} and exceeded the performance of a multi-axis scale used by peer reviewers for the same purpose.⁹ It markedly exceeded the reliability of scoring systems used for awarding National Science Foundation grants.¹⁰ However, it correlated poorly with routine reviewer performance statistics such as rate of recommendation for acceptance and congruence rate. This is logical, since many factors other than just review and manuscript quality affect editorial decisions. It also means that these statistics cannot replace a subjective scale to assess review quality.

The rating scale also showed a modest correlation (R=0.53) with the ability of reviewers to detect deliberate flaws in a fictitious manuscript. The more flaws they were able to detect and report, the more likely they were to produce reviews judged excellent by editors. Reviewers with high average ratings reported twice as many errors as those with low ratings. Reviewers reported only a minority of such flaws, similar to a prior study.¹¹ There is no "gold standard" for judging a quality peer review, but this at least suggests that our subjective quality rating may be related to accuracy and/or thoroughness.

Our scale had only 1 axis and the definition of quality was not specific; previous research has suggested that multiple, more specific scales increase the reliability of ratings.⁹ Despite this, our ICC was higher than previous multi-axis scales used to rate manuscripts,^{7 - 8} although not as reliable as the much more complex scale of Feurer et al,⁵ which included objective data.

We have found this simple scale useful as part of a comprehensive evaluation to identify poor reviewers who should be dropped and excellent reviewers who deserve acknowledgment. Its reliability could be improved by defining the qualities of a good review more specifically and by formally training editors in its use.

In conclusion, this simple 5-point scale for editors to rate peer review quality is acceptably reliable and correlates modestly with reviewers' reporting of flaws in a test manuscript. It provides a useful tool for monitoring reviewer performance and cannot be replaced by routine reviewer statistics such as individual rate of recommendation for acceptance and decision congruence.