Identifying Manuscript Reviewers: Title and subTitle BreakRandomized Comparison of Asking First or Just Sending FREE

In recruiting reviewers, some journals simply send the manuscript to identified experts with a cover letter asking them to review the paper; if unable or unwilling, they may decline ("justsend"). Other journals query potential referees first, and only send manuscripts to those who specifically assent ("askfirst"). We are unaware of evidence as to which is better and authorities provide little guidance. Bishop¹ does not mention the matter specifically, although his statement, "[s]ome journals send out manuscripts for review with a very simple cover letter, ‘Could you please review the enclosed paper as to its suitability for publication in this journal?'" seems to imply justsend. The objective of this study was to compare these 2 approaches.

Setting was the main editorial office of Obstetrics & Gynecology, a monthly medical specialty journal. The editor chose 2 referees for each research article received between September 2, 1999 and May 8, 2000; if fax numbers and mailing address were known for both, the manuscript was enrolled in the study. Using a random-number generator, an editorial assistant assigned one referee to justsend and the other to askfirst.

Referees assigned to justsend were mailed manuscripts and asked to return their reviews by fax or e-mail within 3 weeks; if unable to comply, they were to telephone the editorial office immediately, and a substitute was chosen. Askfirst assignees were faxed information about the manuscript (title, authors, length) and asked to indicate their willingness by return fax. If they accepted, the manuscript and the same instructions and cover letter (asking for a return within 3 weeks) were mailed. If a referee refused or failed to respond within 3 working days, a substitute chosen by the editor was contacted by fax, and so on until a referee agreed to the review (up to 4 cycles) (Figure 1).

Main outcome variables were (1) proportion of initial referees who failed to opt out (justsend) or who opted in (askfirst), (2) overall time for the review process (days from enrolling manuscript in the study until review received), and (3) days between mailing manuscript and receipt of review. Additionally, quality ratings of reviews were compared for the 2 methods, based on a global rating (5-point scale) assigned by the editor at manuscript disposition, without knowledge of assignment for this study. One editor and an editorial assistant were involved.

Sample size, based on assumed 15% opt-out for justsend and assumed 30% failure to opt-in for askfirst (power = 0.80, α = .05), was projected at 300 manuscripts (final sample = 283).

Considering just the initial sample, of 283 justsend referees, 261 (92%) did not decline and 247 (87%) ultimately produced reviews. Of 283 askfirst referees, 181 (64%) agreed to review and 177 (63%) ultimately produced reviews; 43 (15%) declined and 59 (21%) failed to respond. The difference in frequency of specific declines (8% vs 15%) was significant (χ² = 7.66; P = .008). Viewed another way, whereas only 8% of justsend referees opted out, 36% of askfirst candidates did not opt in. However, the rate of producing reviews among initial justsend referees who did not opt out (247/261 [95%]) and askfirst candidates who opted in (177/181 [98%]) did not differ significantly (χ² = 2.72; P = .14).

Considering all reviewers, the mean (SD) overall time required for the review process was similar in askfirst and justsend (24.7 [9.7] vs 25.9[10.5] days; P = .18). However, once the manuscript was mailed to a referee (on day of study enrollment for justsend but not until specific assent was given in askfirst), the time until a review was received was significantly less with askfirst than with justsend (21.0 [9.2] vs 25.0 [10.1 days]; P<.005). Thus, askfirst reviewers tended to complete their reviews sooner, even though all were given the same instructions, and this difference made up for extra time spent in gaining a potential reviewer's assent.

Quality of reviews was assessed in the subset of 155 manuscripts in which both initial referees agreed and returned reviews, because initial assignment was both random and masked, whereas selection of substitute referees could well have been influenced by past quality ratings. There was no significant difference in quality of reviews between justsend and askfirst groups (Wilcoxon sign-rank test, P = .39).

We found, not surprisingly, that making potential referees opt in invited approximately twice as many turndowns as an opt-out approach. Moreover, the substantial segment of unanswered askfirst requests (21%) meant that only 64% were answered affirmatively. Although askfirst had a higher rate of turn-downs, making it necessary to find a substitute referee, the time required for the overall review process was similar to that with justsend. This resulted from askfirst reviewers completing their reviews quicker than justsend reviewers, even though both were allowed the same return time. Perhaps referees assent in advance when they know their schedules will permit them to do the review, whereas those who receive the manuscript without any warning add it to their pile of work and do it when they can. We found no indication that soliciting in advance affected review quality.