Second-Order Peer Review of the Medical Literature for Clinical Practitioners FREE

Clinical journals support several lines of research communication, including scientist-to-scientist (preliminary studies, the predominant mode), scientist-to-clinician (more definitive, ready-for-application original studies), clinician-to-clinician (review articles), and clinician-to-scientist communication (case reports and case series).¹ These communication lines are not clearly differentiated in journals, and content that is ready for direct clinical application by clinicians of a particular clinical discipline is of low frequency, perhaps contributing to the difficulty that clinicians have in keeping up to date.²

Working with medical professional groups, we developed secondary journals and information services, such as ACP Journal Club and Evidence-Based Medicine, and their Web sites, to identify and disseminate the most definitive original studies and systematic reviews. For these secondary journals, research staff select all articles at the time of publication in journals that meet explicit criteria for the critical appraisal of clinical research evidence, and a clinical editorial team selects qualifying studies for attention for a broad range of medical practice, including generalists and specialists.

Physicians reading these secondary publications must select the articles of most relevance to their own practice. To facilitate more exact matching of article content and relevance to individual clinical interests, we developed the McMaster Online Rating of Evidence (MORE) system. MORE collects and collates ratings from practicing clinicians for articles that have passed critical appraisal criteria (Box). MORE now supports the selection and reporting of articles for the secondary publications and also for new services downstream of MORE, including the McMaster Premium Literature Service (PLUS). Through PLUS, alerts about articles are sent to clinicians according to their self-stated discipline(s). In this article we describe the literature selection process and the MORE rating system, which activates a second-order peer review (peer review by clinicians for clinicians), as well as PLUS, a new service supported by MORE.

General

All English-language original and review articles in an issue of a candidate journal are considered for abstracting if they concern topics important to the clinical practice of internal medicine, general and family practice, surgery, psychiatry, pediatrics, or obstetrics and gynecology. Access to foreign-language journals is provided through the systematic reviews, especially those in the Cochrane Library, which summarizes articles from over 800 journals in several languages.

Original articles are classified by purpose (eg, treatment, diagnosis) and then passed or failed on the following methodologic criteria relevant to the purpose of the article. In order to pass, all criteria for the relevant purpose category must be met.

Prevention or Treatment; Quality Improvement

Diagnosis

Prognosis

Causation

Economics of Health Care Programs or Interventions

Clinical Prediction Guides

Differential Diagnosis

Systematic Reviews

*These criteria appear in each of these publications: ACP Journal Club , Evidence-Based Medicine , Evidence-Based Nursing , BMJ Updates+ , and Medscape Best Evidence Alerts .

Research staff trained in health research methodology read all original and review articles at the time of publication (from January 2003-present), over 28 000 articles per year, in each issue (over 1600 issues/year), of over 110 clinical journals (number has varied slightly as new journals were added for regular review or periodically discarded from review if they did not have at least 1-2 articles that met our criteria over a 6- to 12-month period, but number has always been over 110). Articles that were read included a broad range of medical disciplines (http://bmjupdates.mcmaster.ca/journalslist.asp) including primary care, emergency medicine, internal medicine and its subspecialties, general surgery, obstetrics, gynecology, pediatrics, psychiatry, and nursing. Journals were selected based on suggestions by librarians, clinicians, editors, and editorial staff; Science Citation Index (SCI) impact factors; systematic examination of the contents of each selected journal for at least 6 months; and by ongoing yield of articles that meet basic criteria for assessing the quality of studies concerning the cause, course, prediction, diagnosis, prognosis, prevention, and treatment of medical disorders for criteria (Box). For example, criteria for a treatment study are that it must be a randomized trial reporting clinical outcomes, with at least 80% follow-up of participants. Of the 500 or more articles reviewed each week, only about 40 articles pass the methods filter, yielding between 2000 and 3000 articles each year with the strongest methods for clinical attention. The literature selection process is highly reproducible, based on calibration studies previously reported.³ Briefly, after a 1-year period of extensive calibration exercises, 6 research staff were involved in a blinded, inter-rater reliability study and were able to attain chance-corrected agreement coefficients (κ) over 0.80 for all classification areas (ie, article purpose and methods).

More than 400 journal titles have been assessed since 1991 and, based on the number of articles meeting criteria, the top journals form the core set reviewed in producing 3 evidence-based journals, ACP Journal Club, Evidence-Based Medicine, and Evidence-Based Nursing . Journal yield is reviewed annually and journals with very low yield (<1-2 articles/year) are dropped. New journals are added if nominated or if needed to expand the scope of disciplines covered and if they contain articles that meet our criteria based on review of at least 6 journal issues. A detailed description of the literature selection process has been previously published.⁴

We recruited practicing clinicians to rate the usefulness of articles that met our appraisal criteria. The purpose of the rating process was to determine the clinical disciplines for which the topic of each qualifying article was best suited. We estimated that about 1300 raters would be needed to provide adequate discipline coverage and response times. This estimation was based on 40 articles per week that would pass critical appraisal criteria (the methods filter), an average of 3 disciplines assigned to each passing article, the desire to obtain at least 3 ratings for each discipline per article, and the expectation that raters would only want to rate 1 to 2 articles per month (40 articles × 3 disciplines × 3 ratings × 52 weeks/12 months/1-2 articles/month = 780-1560 raters). Raters were recruited according to our target user groups, including primary care physicians and selected specialties (Table 1). Raters were recruited by e-mail, Web site notices, newsletters, and editorials in evidence-based journals.⁵ Selection criteria included: (1) physician in independent clinical practice in 1 or more of the disciplines covered by the service; (2) reliable Internet access, and (3) willingness to respond quickly (within 2 working days) to requests for ratings. Incentives for raters included access to pre-appraised articles in their discipline(s) , continuing medical education credits (15 minutes/article rated), comparison of their ratings with other ratings for the same article, and access to the highest-rated articles in their discipline. Raters registered the maximum frequency with which they could be sent articles for rating.

A system was required to handle the administrative functioning of the rating process. This led to the development of an Internet-based interface to support administrative functions for posting articles that had passed critical appraisal criteria, assigning articles to raters, collecting ratings and comments for each article, and transferring rated articles to the PLUS online delivery system. For each article that was selected based on this selection process, its citation, abstract, and Medical Subject Headings (MeSH) entered a database via an online request to PubMed for the MEDLINE data display using the article's unique PubMed identification number. The retrieved information was parsed into discrete data (eg, title, authors, abstract, MeSH).

The weekly operational cycle of moving articles from the literature selection phase (the hand search described previously) into the MORE system and then readying them for release via PLUS appears in Figure 1. For articles meeting criteria during the literature selection phase, research staff from the Health Information Research Unit assigned topic indexing and clinical discipline codes. All entries were checked by a physician (R.B.H.) then added to the MORE database. E-mail notices of articles to be rated were sent to 3 or 4 practicing physicians (depending on the number of available raters) for each pertinent clinical discipline. Thus, an article on diabetes would be separately rated by 3 to 4 each of family physicians, internists, and endocrinologists. These raters logged into the MORE system directly by clicking on the hyperlinked article title in their notification e-mail. Their individual identification and password were embedded in the link and if determined valid after decryption, the raters were logged onto the MORE system.

Raters reviewed their assigned full-text articles online and rated them on two 7-point scales (highest score, 7) (Figure 2). The first scale, for relevance, concerned the extent to which the article was pertinent to practice in the rater's clinical discipline. If relevance was rated at least 3, the rater completed a second 7-point scale, on the extent to which the article's content represented news or something that clinicians in the rater's discipline were unlikely to know (which we labeled newsworthiness). Optionally, raters also provided up to 1000 characters of free-text comments. Rater's comments that appeared libelous, blasphemous, or thoughtless in the opinion of the MORE staff were not used. Any information that might identify individual raters was excluded.

When at least 3 raters for a clinical discipline had rated an article, the ratings were averaged within that discipline. Articles with mean ratings of 3 of 7 or more for each scale for at least 1 discipline were transferred to the PLUS database and an e-mail alert was sent to PLUS participants registered in that discipline if the ratings met the cutoff set by the user (the default was 5 for each scale). Other disciplines' ratings were added as separate lines to the article record as they became available, and registrants in those disciplines were alerted accordingly. Articles with mean scores of less than 3 of 7 for either scale for all disciplines assigned to the article were transferred to the quarantine database. Articles in the quarantine database therefore passed the methods filter but were considered to be not relevant or informative by practicing clinicians and were not made available to users (Figure 1).

Once in the PLUS database, article records were manually assigned links to their full text in licensed online journals or open-access journals, as available. Links to MedlinePlus about drug prescribing and information for patients were created as available. A link to the PubMed abstract was derived from the article's PubMed identification number and stored in the article record. Access was then provided for registered physician users via the PLUS (a free-access version of this service is now available as BMJUpdates+ [http://www.bmjupdates.com]).

Updates and corrections to each PLUS article's citation, abstract, and indexing data automatically occurred every 2 weeks by repeating the request to PubMed. The PLUS system reporting function identified aberrant data entry with respect to full-text links and patient and drug prescribing links. When alerts were sent to users from the PLUS system, all links displayed in the Web interface were tested for functionality and checked for proper pairing with the citation data.

MORE was established to feed PLUS, an online service that provided physicians with discipline-specific alerts and a search engine for the database of accumulated alerts. An ongoing evaluation of McMaster PLUS is measuring use of, and satisfaction with the service in a randomized clinical trial, which was launched in April 2004. PLUS was created as an adjunct to the Northern Ontario Virtual Library digital library service, and provided free to physician trial participants in Northern Ontario. The Northern Ontario Virtual Library digital library included licensed collections from Ovid Technologies Inc (selected full-text journals, texts, and Evidence-Based Medicine Reviews) and Stat!Ref (a collection of medical texts). The majority of PLUS article citations included links to their Ovid full-text articles or to open-access full-text articles via their publisher. Beginning in November 2003, physicians in Northern Ontario were recruited to participate in the trial by Northern Ontario Virtual Library Website notices, e-mail, fax, and mail invitations. Following online consent, participants registered according to their clinical discipline(s) and since April 2004, were alerted by e-mail to the titles of articles rated through the MORE system for these disciplines. Participants specified the frequency of alerts and adjusted the rating cutoffs according to their preferences, within the range from 4 to 7 for each of the 2 scales. They registered for as many disciplines as they wished, including general practice/family medicine and primary care special interests including anesthesia, mental health, obstetrics, and emergency medicine; internal medicine and its individual subspecialties; obstetrics; gynecology; general surgery; pediatrics; and psychiatry. All user accesses were monitored for linking from article titles in e-mail alerts or search engine hits to the article record in the PLUS database (which included the article's full citation, mean ratings by clinical discipline, and links), and whether the user subsequently accessed links to raters' comments, the PubMed abstract, or if available, the full-text article, drug information and prescribing pages in MedlinePlus, or ACP Journal Club synopsis.

Evaluation of PLUS was reviewed and approved by the McMaster University Hamilton Health Sciences Research Ethics Board.

Rater recruitment for MORE began in 2002, exceeded 1300 in October 2003, and reached 2139 by October 2005. A total of 1042 raters (48.7%) indicated a primary care discipline (at least 1 of general practice, family practice, emergency medicine, or primary care internal medicine) but many indicated multiple interests (Table 1). Just over 57% resided in North America and 25% in the European Union, with all continents represented (Table 2). Approximately 74% of individuals registered at the end of October 2005 had rated an article and the number of articles rated ranged from 1 to 98. The number of raters for an article ranged from 3 (for an article with only 1 discipline) to 43 (for an article with multiple disciplines). Raters who did not respond to repeated requests to rate the literature were retired (286 by October 2005; 13.4% of the rating pool). To date, we have not compared raters who responded to rating requests with raters who did not.

For articles published in 2004 (the most recent complete year), ratings (Figure 2) had a mean score of 5.42 for relevance to practice and 4.65 for useful new information (newsworthiness) (Table 3). On average, primary care physicians rated articles lower than specialists for both relevance and new information (Table 3). Systematic review articles were rated higher than original articles for relevance, but lower for useful new information (newsworthiness) (Table 3).

The distribution of ratings for articles for 2004 (Table 4) showed that 24.9% of articles that met critical appraisal criteria were rated at 6 or above for relevance, and fewer (6.2%) were rated at 6 or above for useful new information (newsworthiness). Thus, a primary care user of the PLUS system who had set the cutoffs for sending alerts at 7 of 7 for relevance and 6 of 7 for useful new information (newsworthiness) would have been sent e-mail notification of 40 articles published in 2004 (0.14% of the 28 737 articles with abstracts screened that year). Just 40 (1.8%) of the articles passing critical appraisal for 2004 received mean ratings of less than 3, the cutoff for the quarantine database.

From November 2003 to April 2004, 98 physicians working in Northern Ontario were recruited to receive alerts from PLUS. From April 2004 through September 2005, alerts were sent to at least 1 participant for 2698 newly processed articles, for a total of 49 353 alerts, Thus, alerts about a given article were sent to about 18 participants on average, according to their registered clinical discipline(s). Participants were alerted by e-mail about 6 to 1931 articles per month, a mean of 28.7 per month (95% CI, 24.7-32.7). During an 18-month trial period, 88 (90%) used the service logging into PLUS 3136 times. Of these, 1784 (56.9%) logins were to seek information and 1352 logins (43.1%) were in response to article alerts. Just over 30% of participants used the service at least twice per month. Participants responded to a mean of 14.7% of e-mail alerts (95% CI, 10.6%-18.9%), with a range per participant from 0.1% to 85.4%. During this period, participants accessed 3137 article records (full citation, ratings, comments, and links), 2018 article abstracts, 742 full-text articles, and 317 rater comments.

A chilling view of the Internet era warns that high-quality information will inevitably be fully overwhelmed by low-quality information because the cost of production of low-quality information is less and the marginal cost of distribution of information on the Internet is close to zero.⁶ Recent studies show that several barriers block the acquisition and use of high-quality information in health care. Physicians' daily routines afford them little time to search and review new medical information.^7- 9 Physicians admit that they lack the skills required to navigate literature databases^9- 11 and to properly appraise medical literature.^8,11- 12

To try to help overcome these problems, we created a system that centralizes the basic processes of critical appraisal and clinical relevance rating and then channels articles to physicians according to their practice disciplines. For 28 737 original and review articles published in over 110 journals in 2004, we identified 2237 (7.8%) articles for clinical attention, with just a small percentage of these at the highest level of interest for a given clinical discipline. The clinical peer rating system revealed differences in the perceptions of primary care physicians and specialists concerning articles, including articles of potential interest to both groups, as well as differences in ratings for original articles compared with review articles. Of the 98 physicians who registered for PLUS, 10 (10%) did not make use of it. Some of the nonresponse was likely due to technical problems including firewalls, junk mail filters, slow dial-up Internet access, and plain text e-mail defaults (the alert links required that e-mail clients permit Hypertext Markup Language formatting), but we have not formally assessed this. Participants who responded to e-mail alerts (which contained only the titles of articles) clicked through to article records in the PLUS database with ratings only about 15% of the time, and clicked through to article abstracts much more often than to full-text articles. Article ratings for both primary care physicians and specialists were correlated with access rates (data not shown), suggesting that this information influenced the decisions of users concerning which articles to review.

The PLUS system includes rated articles from the beginning of 2003, has ongoing input of articles, and various user interfaces are currently being evaluated in a series of randomized trials. The scope of PLUS is expanding to include more clinical disciplines including selected subspecialties of surgery and nursing disciplines. We continue to recruit raters for all disciplines. MORE ratings are used to help select content for evidence-based journals, an evidence-based textbook of internal medicine (http://pier.acponline.org/index.html), and 2 free alerting services (www.bmjupdates.com and https://profreg.medscape.com/px/newsletter.do). The direct cost of developing the MORE rating system, including software and recruiting, was approximately Can $180 000, and supported by grants (see Acknowledgements section), with ongoing annual cost of about Can $80 000, fully supported by the sponsors of the evidence-based products it now feeds. Raters' time is voluntarily contributed but rewarded in kind (see Methods section). Our ultimate aim is to provide high-quality, clinically rated articles from MORE for all clinicians interested in keeping up to date with important new health care knowledge and all health care publications purporting to be evidence-based. By sharing the costs across many publications, we may be able to compete with lower-quality information on the Internet, challenging Coiera's bleak vision.⁶

Several limitations must be considered for our demonstration project to date. Our preliminary evaluation was pitched at the level of feasibility and usability testing and the data are observational. The raters were volunteers and their ratings may not be representative of those of most physicians. Articles being rated were not masked for authors and journal, and all corresponding editorials were sent along with the article being rated. The potential effect that this may have on the ratings has not been assessed. However, we thought that this information should be taken into account by raters. Our rating scales may not be optimal for reliability or their ability to discriminate between the interests of various clinical disciplines or convey useful information to users. These matters warrant further study, but our preliminary data suggest that the scales do discriminate among disciplines and study types and influence users' decisions about which articles to read.

The PLUS service was piloted in Northern Ontario because of the availability of project funds: the Ministry of Health and Long-term Care in Ontario wanted to build information access for physicians in Northern Ontario to enhance opportunities for continuing education and prepare for a new medical school. This was auspicious for our purposes because the region is served by a digital health library, the Northern Ontario Virtual Library, which permitted linking of alerts to their full-text articles. We do not know how adaptable this experience is to other locations and settings with higher population density. Whether participants learned and applied the information they reviewed is the subject of ongoing evaluation of the PLUS system; no assessment is currently under way to assess effects on clinician performance or patient outcomes. Thus, little basis exists at present for assessing whether the differences in article ratings by discipline, or by type of article (original vs review), are important, especially in terms of the overall objective of improving practitioner performance and patient care.

Nevertheless, clinical ratings may be useful in facilitating continuing education activities by identifying the most sound, relevant, and sought-after new studies and reviews, alerting clinicians to articles describing critically appraised studies deemed of relevance and interest by their peers.