0
Peer Review Congress |

Reviewing the Reviews:  The Example of Chronic Fatigue Syndrome FREE

John Joyce, MB, BCh, MRCPsych; Sophia Rabe-Hesketh, PhD; Simon Wessely, MA, BM, BCh, MSc, MD, FRCP, MRCPsych
[+] Author Affiliations

From the Institute of Psychiatry (Drs Joyce, Wessely, and Rabe-Hesketh), King's College School of Medicine, King's College, and Maudsley Hospitals (Dr Wessely), London, England.


JAMA. 1998;280(3):264-266. doi:10.1001/jama.280.3.264.
Text Size: A A A
Published online

Objective.— To test the hypothesis that the selection of literature in review articles is unsystematic and is influenced by the authors' discipline and country of residence.

Data Sources.— Reviews in English published between 1980 and March 1996 in MEDLINE, EMBASE (BIDS), PSYCHLIT, and Current Contents were searched.

Study Selection.— Reviews of chronic fatigue syndrome (CFS) were selected. Articles explicitly concerned with a specialty aspect of CFS and unattributed, unreferenced, or insufficiently referenced articles were discarded.

Data Extraction.— Record of data sources in each review was noted as was the departmental specialty of the first author and his or her country of residence. The references cited in each index paper were tabulated by assigning them to 6 specialty categories, by article title, and by assigning them to 8 categories, by country of journal publication.

Data Synthesis.— Of 89 reviews, 3 (3.4%) reported on literature search and described search method. Authors from laboratory-based disciplines preferentially cited laboratory references, while psychiatry-based disciplines preferentially cited psychiatric literature (P=.01). A total of 71.6% of references cited by US authors were from US journals, while 54.9% of references cited by United Kingdom authors were published in United Kingdom journals (P=.001).

Conclusion.— Citation of the literature is influenced by review authors' discipline and nationality.

Figures in this Article

MANY narrative reviews and reviews that describe themselves as systematic have been shown to be nonreproducible and to be of low mean scientific quality.13 A lack of clearly specified methods of identifying, selecting, and validating included information has been among the problems noted.4 Experts could not agree, even among themselves, about whether other experts who wrote review articles had conducted a competent search or generated a bias-free list of citations.5,6

Few things are certain about chronic fatigue syndrome (CFS) other than that it is controversial. Both public and professional opinions are often debated passionately. In such circumstances both physicians and interested members of the public may turn for guidance and information to review articles. Such articles fulfill an important function for professionals, journalists, and patients unable to find, locate, or evaluate primary sources of information. This is particularly important in CFS, since potentially relevant research spans many disciplines, with important contributions coming from specialties as diverse as immunology, virology, internal medicine, psychiatry, psychology, and neurology. Thus, our aim was to examine the quality of current reviews of CFS. Our hypothesis was that use of the literature would show the following biases: the identification and selection of literature for review is unsystematic, it fails to reflect the broad range of literature, and it is influenced by the author's discipline and country of residence.

Data Sources

All reviews of CFS between 1980 and 1996 from English-language journals were eligible. We defined a review as an article that made a claim, either implicit or explicit, referring to the range of knowledge known at the time of publication and that represented itself as being able to reach general conclusions about CFS. Articles explicitly labeled as dealing with a specialty aspect of CFS, such as "psychiatric aspects of CFS" or "immunological findings in CFS," were excluded. Reviews with fewer than 15 references (we considered a citation list of at least 15 as necessary evidence of a serious attempt to review the subject) or without any details about authors were excluded. Seventy-three foreign language reviews in 14 different languages found in the same search were excluded due to lack of linguistic expertise and small numbers per country of journal publication.

We searched MEDLINE, EMBASE (BIDS), PSYCHLIT and Current Contents. We conducted a free text search using the terms chronic fatigue syndrome, neurasthenia, myalgic encephalomyelitis, and tiredness and the truncated terms chronic fatigue and postviral. More than 4000 references were checked in title and abstract by one of the authors (J.J.). All possible reviews were then confirmed by one of the authors (S.W.).

Data Extraction

We used 4 phases in extracting and categorizing data. First, using the first 3 of 10 criteria for the assessment of scientific quality of research overviews and how they relate to selecting literature to reviews as recommended by Oxman and Guyatt,7 we noted comments the authors made on their search methods, each article's comprehensiveness, and the review article's inclusion criteria. Second, we divided the tasks of tabulating (J.J.) and checking (S.W.) between the departmental specialty of the first author and his or her country of residence and found no discrepancies. Third, the references cited in each index paper were tabulated and assigned to 6 specialty or subject categories by article title alone as shown in Table 1. Fourth, the references cited were also tabulated and assigned to 8 categories representing the countries in which they were published, including the United States, United Kingdom, Europe (excluding the United Kingdom), Australia, Canada, New Zealand, other (which included South Africa, Israel, India, Japan, and China), and not listed. The place of publication of each journal title was ascertained by consulting Libertas' list of serials.

Table Graphic Jump LocationTable 1.—References Tabulated According to Specialty by Article Title
Statistical Methods

The data consist of a set of percentages for each review, representing the number of references that fall into each subject or country of publication. We wished to test how the author's discipline, country of residence, and the country in which the article was published had affected the use of references. Since the subject or country of publication categories may be regarded as repeated measurements within each review and the data are approximately normally distributed, this could be done by repeated-measures multivariate analysis of variance (MANOVA) using the matrix of percentages. In order to display the multivariate data in 2 dimensions, we made a series of biplots,8 ie, plotting the first 2 principal components of the matrix of percentages together with a set of axes for the reference subject or country of publication category. The repeated MANOVA was carried out using SPSS statistical software (SPSS Inc, Chicago, Ill) and the biplots were made using S-Plus (MathSoft, Seattle, Wash).

One hundred eighty-six reviews were found on our preliminary search. Eighty-nine of these were counted as eligible for analysis. All were checked by one of the authors (S.W.) and there were no disagreements between the 2 reviewers. Fifty-five of the articles concerned a specialty aspect of CFS, despite a general title, or were primary research, an audit, a case report, or a first-person account (26 of these were reviews specifically dealing with treatment only). Thirteen were not attributed to an author, 7 were without any references, and 18 had fewer than 15 references. Four were unobtainable despite a thorough interlibrary search.

Details of Literature Search

Only 3 (3.4%) of the 89 reviews reported on the database source or sources used to conduct its literature search, none of which were written by any of the current authors. In 2 articles, the authors merely described the search, without further elaboration, as "relevant published research literature," and in 1 article, the author reported the databases used for the search. One of the 3 specifically reported its inclusion criteria.

Reference Disciplines by Specialty of Author

After data inspection, 1 review (by a neurologist) was excluded as an outlier, as its large distance from all other data would exert a disproportionate influence.

Because of the insufficient numbers of nurse-therapists, pharmacologists, and those in the other group for testing, we avoided the problems of inference, which would be caused by having too many small groups, by including for analysis only those groups with a number of authors who were mostly physicians, either general practitioners or specialized in infectious diseases or psychiatry. There was a highly significant interaction between reference disciplines and specialty of author (P=.01, F12,185=2.84). Figure 1 shows the pattern of reference use by infectious diseases specialists and psychiatrists. Those working in infectious diseases quote the laboratory category most often followed by physicians, nurses, and general practitioners. Infectious diseases specialists and general practitioners quote psychiatric articles least. Psychiatrists and pharmacologists quote the laboratory category and psychiatric categories about equally.

Graphic Jump Location
Figure 1.—Use of reference disciplines by infectious disease specialists and psychiatrists.

The biplot in the left panel of Figure 2, shows the spread of reviews by specialty of author, in this case psychiatrist or infectious diseases specialist according to reference disciplines. Those reviews authored by infectious diseases experts cluster on the left around the laboratory category, and those authored by psychiatrists cluster on the right around general psychiatry. Principal components accounted for 73.5% and 14.4% of the variance.

Graphic Jump Location
Figure 2.—Left, Biplot of reference disciplines by specialty of author. Right, Biplot of country in which reference was published and in what country the authors reside. The points represent reviews, and distances between points reflect differences between reviews in their citation patterns, according to discipline (left) and country (right). The axes show the direction in which the percentage of references in a reference category increase, and tick marks represent 5% increments. For example, the left side of the figure shows that reviews vary from approximately 0% references in the laboratory category to 100% from right to left.
Country of Residence and Reference Selection

The distribution of countries of reference publication was compared by countries where the review author resided using repeated measures-MANOVA. Only references from the United States and United Kingdom were included due to small numbers of articles published in other countries. The interaction between where the author lived and where the reference was published was significant (P<.001, F1,72=145.3).

The 2 principal components accounted for 92.1% and 5% of the variance, respectively (97.1% in total). The right panel of Figure 2 shows the spread of reviews according to where the author resides and in what country the references were published. Those reviews with US authors cluster to the right and those with United Kingdom authors cluster to the left. We have not included the 6 reviews by Australian authors in this biplot in order to preserve the clarity of presentations. However, Table 2 shows that the Australian reviews were situated midway between the United States and the United Kingdom in terms of citation distribution by country.

Table Graphic Jump LocationTable 2.—Mean Percentage of Reference Country of Publication Across All Reviews and by US, United Kingdom, and Australian Authors

Despite the recent emphasis on the necessity for quality in medical reviews, our results show that in the area of CFS the vast majority of reviews are not based on systematic literature searches and do not use objective criteria for inclusion or exclusion.

One might reasonably expect that reviews of a multidisciplinary subject, such as CFS, are able to integrate findings from many sources–instead it is possible that they perpetuate preexisting disciplinary biases. We have shown that the choice of articles to cite is influenced by the author's discipline and the country in which he or she resides. We emphasize that we have only looked at review articles that claimed to be comprehensive and have not included any review articles that made such a bias explicit. A reader consulting any of the review articles we have studied expects them to be an objective synthesis of a complex subject. Instead, most display biases toward particular disciplines, usually the one in which the author practices. Reference bias has been previously reported in drug trials9 but not, to our knowledge, in the area of reviews. Such biases are not unexpected but are important. Similarly, it is a staple of academic gossip that Americans only cite US literature, Europeans European literature, and so on. We have confirmed that both US and United Kingdom authors are more likely to cite literature published in their own countries. Also of note is the underuse of Continental European literature by US or United Kingdom authors. We are aware of only 1 previous confirmation of this intuition.10

Exclusion of references because of language has been shown to introduce bias in randomized controlled trials.1113 We acknowledge our findings can only be generalized to the English-language literature. For fairly obvious reasons, language bias alone cannot explain our findings. American English and United Kingdom English may not sound the same, but they do read the same.

Dawson NV, Arkes HR. Systematic errors in medical decision making: judgment limitations.  J Gen Intern Med.1987;2:183-187.
Jadad AR, McQuay HJ. Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology.  J Clin Epidemiol.1996;49:235-243.
Chalmers TC, Frank CS, Reitman D. Minimizing the three stages of publication bias.  JAMA.1990;263:1392-1395.
Mulrow C. The medical review article: state of the science.  Ann Intern Med.1987;106:485-488.
Oxman AD, Guyatt GH, Singer J.  et al.  Agreement among reviewers of review articles.  J Clin Epidemiol.1991;44:91-98.
Ingelfinger FJ. Peer review in biomedical publication.  Am J Med.1974;56:686-692.
Oxman AD, Guyatt GH. The science of reviewing research.  Ann N Y Acad Sci.1993;703:125-133.
Gower JC, Hand DJ. Biplots . London, England: Chapman & Hall; 1996.
Gotzsche PC. Reference bias in reports of drug trials.  Br Med J (Clin Res Ed).1987;295:654-656.
Campbell F. National bias: a comparison of citation practices by health professionals.  Bull Med Libr Assoc.1996;78:376-382.
Gregoire G, Derderian F, Le Lorier J. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias?  J Clin Epidemiol.1995;48:159-163.
Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomized controlled trials published in English and German.  Lancet.1997;350:326-329.
Moher D, Fortin P, Jadad AR.  et al.  Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews.  Lancet.1996;347:363-366.

Figures

Graphic Jump Location
Figure 1.—Use of reference disciplines by infectious disease specialists and psychiatrists.
Graphic Jump Location
Figure 2.—Left, Biplot of reference disciplines by specialty of author. Right, Biplot of country in which reference was published and in what country the authors reside. The points represent reviews, and distances between points reflect differences between reviews in their citation patterns, according to discipline (left) and country (right). The axes show the direction in which the percentage of references in a reference category increase, and tick marks represent 5% increments. For example, the left side of the figure shows that reviews vary from approximately 0% references in the laboratory category to 100% from right to left.

Tables

Table Graphic Jump LocationTable 1.—References Tabulated According to Specialty by Article Title
Table Graphic Jump LocationTable 2.—Mean Percentage of Reference Country of Publication Across All Reviews and by US, United Kingdom, and Australian Authors

References

Dawson NV, Arkes HR. Systematic errors in medical decision making: judgment limitations.  J Gen Intern Med.1987;2:183-187.
Jadad AR, McQuay HJ. Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology.  J Clin Epidemiol.1996;49:235-243.
Chalmers TC, Frank CS, Reitman D. Minimizing the three stages of publication bias.  JAMA.1990;263:1392-1395.
Mulrow C. The medical review article: state of the science.  Ann Intern Med.1987;106:485-488.
Oxman AD, Guyatt GH, Singer J.  et al.  Agreement among reviewers of review articles.  J Clin Epidemiol.1991;44:91-98.
Ingelfinger FJ. Peer review in biomedical publication.  Am J Med.1974;56:686-692.
Oxman AD, Guyatt GH. The science of reviewing research.  Ann N Y Acad Sci.1993;703:125-133.
Gower JC, Hand DJ. Biplots . London, England: Chapman & Hall; 1996.
Gotzsche PC. Reference bias in reports of drug trials.  Br Med J (Clin Res Ed).1987;295:654-656.
Campbell F. National bias: a comparison of citation practices by health professionals.  Bull Med Libr Assoc.1996;78:376-382.
Gregoire G, Derderian F, Le Lorier J. Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias?  J Clin Epidemiol.1995;48:159-163.
Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomized controlled trials published in English and German.  Lancet.1997;350:326-329.
Moher D, Fortin P, Jadad AR.  et al.  Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews.  Lancet.1996;347:363-366.
CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 30

Related Content

Customize your page view by dragging & repositioning the boxes below.