Discussion Sections in Reports of Controlled Trials Published in General Medical Journals:  Islands in Search of Continents? FREE

SEVERAL MAJOR health care journals have already adopted the Consolidated Standards of Reporting Trials (CONSORT) recommendations to make it easier for readers to assess the quality of controlled trials.¹ This is the first joint attempt by biomedical journals to improve the quality of reports of controlled trials, a topic of research for 4 decades.² No other category of biomedical report has received such sustained attention, and this reflects the practical importance of controlled trials in guiding decisions in health care.

Previous research has highlighted deficiencies in descriptions of the materials and methods used and the analysis and presentation of results.² Like most similar articles before it, the CONSORT statement concentrates on these 2 elements of reports of controlled trials. By contrast, the quality of introduction and discussion sections in trial reports has received little systematic scrutiny. The typical discussion section usually addresses a number of dimensions, but, crucially, it is in this section that readers will look for an answer to Bradford Hill's "bottom line" question for any research article: "What does it mean, anyway?"³ This was recognized in the CONSORT statement, which included the recommendation that trialists should "state general interpretation of the data in light of the totality (our emphasis) of the available evidence."¹

Other research has illustrated how selective citation of previous research in the discussion sections of research articles can be biased. Studies that have yielded relatively dramatic results are more likely to be cited in reports of subsequent similar studies than previous studies yielding unremarkable point estimates of effects.⁴ In addition, authors from a particular country or specialty have been shown to selectively cite material generated from within that country^5- 6 or specialty.^7- 8

Ideally, the discussion section of the report of a new trial should involve the presentation of an up-to-date systematic review,⁹ as was done, for example, in the 1986 article on the First International Study of Infarct Survival (ISIS-1).¹⁰ To show how closely reports of controlled trials reflect this ideal more than a decade after ISIS-1, we assessed the discussion sections of all reports of randomized trials published in May 1997 in 5 general medical journals. This month was chosen to allow as up-to-date an assessment as possible and without prior knowledge of thearticles to be published. We concentrated on how well the discussion sections in these articles had made a systematic attempt to set the results of the reported trial in the context of the totality of the other relevant existing evidence.

An article was eligible for inclusion as a trial if it met the following criteria: (1) It was published during May 1997 as a full article (ie, not in the editorials, news, or correspondence sections of the journals) in 1 of the following journals: Annals of Internal Medicine, BMJ, JAMA , The Lancet , and The New England Journal of Medicine (all except the last of these journals have adopted the CONSORT statement). (2) On the basis of the best available information, "the individuals (or other units) followed in the trial were assigned prospectively to one of 2 (or more) alternative forms of health care using random allocation or some quasi-random method of allocation (such as alternation, date of birth, or case record number)," that is, randomized and quasi-randomized trials, as defined by the Cochrane Collaboration.¹¹

If the discussion section of an eligible article contained the results of an attempt to identify and consider all other similar trials (if an attempt either was or was not made to combine their results quantitatively with those of the newly reported trial), it was classified as a systematic review. Relevant issues of the 5 journals were ordered using a list of random numbers. One of the authors (M.C.) searched the issues in this order and the other (I.C.) searched them in reverse order. They independently identified articles that met the eligibility criteria.

The authors independently assessed the discussion section of each eligible article to decide whether an attempt had been made to integrate the results of the new trial into a systematic review, either qualitatively or quantitatively. Such a review could have been either done previously or done especially for the trial article by the authors of that article.

Disagreements on both assessments were resolved by discussion between the authors. Any eligible articles that were initially identified by only one of the authors were read and assessed for the inclusion of systematic reviews by the other before this aspect was discussed.

Reports of other trials that seem to have addressed the question concerned in the index article were sought by searching the Cochrane Controlled Trials Register.¹² Advice was also sought from others with content expertise.

Table 1 shows the number of issues of each of the 5 selected journals in May 1997 and the distribution of the articles identified.^13- 38 Twenty-six articles were identified that met the eligibility criteria, and all articles were identified by both of us independently. One additional article was identified by one of us (M.C.) as possibly eligible, but was rejected following discussion.

Six of the 26 eligible articles claimed to report the first trial addressing a particular question. However, following a search of the Cochrane Controlled Trials Register and discussions with the relevant Cochrane Collaborative Review Groups, similar trials were identified for 5 of the 6 articles. It was not our intention to do a systematic search for trials or to make judgments about whether there was sufficient similarity between the reported trial and previous trials to combine them in a formal meta-analysis. Instead, we simply tried to identify studies that should be considered for inclusion in a systematic review. On this basis, we judged only 1 of the 26 trials identified as likely to genuinely be the first to address a particular question.³⁷

Of the remaining 25 articles, 2 discussed the results of the reported trial in the context of a systematic review. Four others mentioned prior systematic reviews but did not attempt to integrate the new results into updated versions of these reviews. Many of the other articles, including some that claimed to be the first trial addressing a particular question, did contain citations to previous trials, but it is unclear whether these represented all similar trials, how they had been identified, or why they had been included.

We have attempted to find studies similar to our own in which others have assessed whether trial reports set their results in the context of the totality of the evidence; so far, we have been unsuccessful. We recognize that our study can provide nothing more than a snapshot of the current state of the discussion sections of trial reports. However, the articles we assessed were published in 5 widely respected journals, 4 of which have endorsed the CONSORT recommendations. It was not our aim to assess whether the discussion sections of trial reports have improved or worsened over time. Rather we sought to assess whether a problem existed at the time of our study. We suggest that our findings confirm a problem did exist in May 1997. We welcome other studies, since these would help to assess the extent of the problem.

We did not investigate how well the authors of a trial report fulfilled the other important role of the discussion section as an opportunity to summarize and critique the findings of their own trial.³⁹ Others have done this within particular health care questions. For example, in a study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis, Rochon and colleagues⁴⁰ found the data presented did not support conclusions about efficacy in nearly one fifth of the articles, and did not support claims about toxicity in about half the articles. We did not investigate whether each article's introduction section included details of a systematic review done in advance of the reported research. Any such review was likely to be out of date by the time the new trial was completed and published. If a review had been done in advance of the new trial, then it would seem sensible to update it in the discussion section.

More than 10 years after the ISIS-1 article, considerable scope remains for authors and journals to help readers set the results of new trials in the context of previous research. Very occasionally, a new study can be confidently regarded as unique and the manuscript can thus justifiably be regarded as providing the totality of evidence addressing a particular question. However, these circumstances are likely to be rare, and such claims need to be justified by a thorough search for earlier studies. Smith and Goodman⁴¹ concluded that the failure of authors to mention relevant previous work is probably a widespread form of research malpractice. They commended the suggestion that, when submitting reports for journal review, investigators should provide evidence that they have made a thorough search for relevant previous work.⁷

An editorial accompanying the 1986 ISIS-1 article referring to the updated systematic review of relevant data presented in its discussion section acknowledged that "there is a good case for such analyses." But it went on to state that if anyone suggested these should become a regular feature of clinical trial reports, The Lancet would "lead the opposition."⁴² There seems less justification for such a response 12 years later with the development of electronic publication. For example, 1 of the 2 articles identified in this study as containing an updated systematic review (both of which were published in The Lancet) summarized the findings of that review and referred readers to the electronically published versions of the relevant reviews.^23,43- 44

The best way to report the findings of a new trial in the context of earlier studies would be to include an updated systematic review in the discussion section. This would reduce the tendency for the new trial or a select group of previous trials to be viewed in isolation from the whole body of relevant previous research. If, in planning their study, the trialists had to identify and consider relevant prior research, then the production of a systematic review to incorporate their eventual results should be much easier. In addition, with the rapidly increasing number of systematic reviews included in the Cochrane Library,⁴⁵ trialists should be increasingly able to find a ready source for the context into which to set their results.

The public is often confused by the conflicting messages it receives as a result of piecemeal reporting of research. To deserve the public's continued support and trust, researchers and journals need to ensure that reports of research end with scientifically defensible answers to Bradford Hill's question, "What does it mean, anyway?" Those who turn to reports of trials to help guide treatment deserve nothing less than a discussion of the totality of the relevant evidence, as rightly recommended by the CONSORT Group. To paraphrase John Donne, No trial is an island, entire of itself; every trial is a piece of the continent, a part of the main.⁴⁶