0
We're unable to sign you in at this time. Please try again in a few minutes.
Retry
We were able to sign you in, but your subscription(s) could not be found. Please try again in a few minutes.
Retry
There may be a problem with your account. Please contact the AMA Service Center to resolve this issue.
Contact the AMA Service Center:
Telephone: 1 (800) 262-2350 or 1 (312) 670-7827  *   Email: subscriptions@jamanetwork.com
Error Message ......
Brief Report |

Use of the CONSORT Statement and Quality of Reports of Randomized Trials:  A Comparative Before-and-After Evaluation FREE

David Moher, MSc; Alison Jones, BA; Leah Lepage, PhD; for the CONSORT Group
[+] Author Affiliations

Author Affiliations: University of Ottawa (Mr Moher) and Thomas C. Chalmers Centre for Systematic Reviews (Mr Moher, Ms Jones, and Dr Lepage), Ottawa, Ontario.


JAMA. 2001;285(15):1992-1995. doi:10.1001/jama.285.15.1992.
Text Size: A A A
Published online

Context The Consolidated Standards for Reporting of Trials (CONSORT) statement was developed to help improve the quality of reports of randomized controlled trials (RCTs). To date, a paucity of data exists regarding whether it has achieved this goal.

Objective To determine whether use of the CONSORT statement is associated with improvement in the quality of reports of RCTs.

Design and Setting Comparative before-and-after evaluation in which reports of RCTs published in 1994 (pre-CONSORT) were compared with RCT reports from the same journals published in 1998 (post-CONSORT). We included 211 reports from BMJ, JAMA, and The Lancet (journals that adopted CONSORT) as well as The New England Journal of Medicine (a journal that did not adopt CONSORT and was used as a comparator).

Main Outcome Measures Number of CONSORT items included in a report, frequency of unclear reporting of allocation concealment, and overall trial quality score based on the Jadad scale, a 5-point quality assessment instrument.

Results Compared with 1994, the number of CONSORT checklist items in reports of RCTs increased in all 4 journals in 1998, and this increase was statistically significant for the 3 adopter journals (pre-CONSORT, 23.4; mean change, 3.7; 95% confidence interval [CI], 2.1-5.3). The frequency of unclear reporting of allocation concealment decreased for each of the 4 journals, and this change was statistically significant for adopters (pre-CONSORT, 61%; mean change, −22%; 95% CI, −38% to −6%). Similarly, 3 of the 4 journals showed an improvement in the quality score for reports of RCTs, and this increase was statistically significant for adopter journals overall (pre-CONSORT, 2.7; mean change, 0.4; 95% CI, 0.1-0.8).

Conclusion Use of the CONSORT statement is associated with improvements in the quality of reports of RCTs.

Reports of randomized controlled trials (RCTs) are the "gold standard" by which health care professionals and others make decisions about treatment effectiveness. To assess the strengths and limitations of RCTs, readers need and deserve to know the quality of their methods. Previous studies1,2 indicate that reports of low-quality RCTs, compared with reports of higher-quality ones, overestimate the effectiveness of interventions by about 30% across a variety of health care conditions.

The Consolidated Standards for Reporting of Trials (CONSORT) Group developed the CONSORT statement,3 an evidence-based approach to help improve the quality of reports of RCTs. Since its publication in 1996, the CONSORT statement has been widely supported,46 has been translated into several languages, and has an Internet presence (http://www.consort-statement.org) to facilitate awareness and dissemination. However, there is a paucity of data7 regarding whether the CONSORT statement has improved the quality of reports of RCTs.

We compared a sample of RCTs published in 1994 (pre-CONSORT) to a sample published in 1998 (post-CONSORT). To be included, RCT reports had to be drawn from journals classified by the Science Citation Index as "general and/or internal medicine." Four journals were evaluated: BMJ, JAMA, The Lancet, and The New England Journal of Medicine (NEJM). Three of these journals (BMJ, JAMA, Lancet) were early adopters of the CONSORT statement,46 whereas the fourth (NEJM) did not formally adopt the CONSORT statement and was used as a comparator.

Reports were included only if they involved human subjects and if the study design was identified as an RCT by examining the title and the abstract. Two readers (A.J., L.L.) independently hand-searched the 4 journals for RCTs published between January 1 and June 30 of both 1994 and 1998. Hard copies of relevant articles were obtained but were not masked because evidence concerning the effect of masking on assessments of trial quality is inconsistent.8,9

Three measures were used to assess the quality of reports of RCTs. First, the CONSORT checklist was modified so that multiple items were listed separately, which resulted in 40 items. Each item was assigned a yes or no response depending on whether the authors had reported it. Second, the reporting of allocation concealment was assessed as adequate, inadequate, or unclear.2 Third, the Jadad scale,9 which contains 2 questions for randomization and masking and 1 question evaluating the reporting of withdrawals and dropouts, was used to assess quality. Each question entails a yes or no response option. In total, 5 points can be awarded, with higher scores indicating superior quality. Two reviewers (A.J., L.L.) completed all of these evaluations.

Both reviewers underwent training in evaluating RCTs using the CONSORT checklist. Before training, the definition of each checklist item was discussed. To assess interobserver agreement, 5 items from the checklist were purposefully selected (inclusion criteria, exclusion criteria, point estimate, deviation from protocol, and general interpretation of study findings). A κ statistic was calculated for each item based on a randomly selected set of 10 RCTs, from 1994 and 1998, and these were not included in this study. Discrepancies were resolved by consensus involving a third party (D.M.). A similar approach was used to assess interobserver agreement in assessing reporting of allocation concealment and using the Jadad scale.

Taking time (1994-1998) into consideration and using either unpaired t tests or χ2 tests, we compared the number of checklist criteria included in each report and the mean number of criteria included within each subheading specified in the CONSORT checklist. We also assessed the percentage of studies that reported unclear allocation concealment and the specific item and overall quality score derived from the Jadad scale. The number of CONSORT checklist items reported was treated as the dependent variable in an analysis of variance (ANOVA) with 2 factors: journal and year. The difference between the adopter journals and the comparator journal was derived from the fitted ANOVA using a contrast involving journal (ie, adopters vs comparator) and year (ie, 1998 vs 1994). Least-square estimates and 95% confidence intervals (CIs) also were computed; for significant level, P = .05 (2-sided).

Of the 221 RCTs identified, 211 met the inclusion criteria. Six studies were excluded because they were animal studies and 4 studies because they were quasi-randomized trials.

Substantial agreement was established for 4 items from the CONSORT checklist and the Jadad instrument (inclusion and exclusion criteria, point estimate and general interpretation [κ = 1.0], quality assessment, and overall Jadad score [κ = 0.74]). Moderate agreement was established for allocation concealment (κ = 0.53) and for 1 item from the CONSORT checklist (κ = 0.54, deviation from protocol).

There was an increase over time in the number of CONSORT checklist items included in the reports of RCTs in all 4 journals (Table 1). This increase was statistically significant for 2 individual journals and overall for adopter journals (pre-CONSORT, 23.4; mean change, 3.7; 95% CI, 2.1-5.3). Over time, the increase in the reporting of CONSORT items was significantly greater for adopter journals when evaluated against the comparator journal (mean difference, 3.8; 95% CI, 1.0-6.5; 2-sided P = .007).

Table Graphic Jump LocationTable 1. CONSORT Checklist Criteria Included in Reports of Randomized Trials for Articles Published in BMJ, JAMA, The Lancet, and The New England Journal of Medicine (NEJM) During the First Half of 1994 and 1998*

The proportion of RCTs with unclear reporting of allocation concealment decreased over time in all 4 journals (Table 2) and was statistically significant for adopter journals (pre-CONSORT, 61%; mean change, −22%; 95% CI, −38% to −6%). Over time, 3 of the 4 journals improved the quality of reports of RCTs as assessed by the Jadad scale (Table 2), which was statistically significant for 1 journal (Lancet) and across the adopter journals (pre-CONSORT, 2.7; mean change, 0.4; 95% CI, 0.1-0.8).

Table Graphic Jump LocationTable 2. Quality of Reports of Randomized Trials, Using an Assessment Tool, for Articles Published in BMJ, JAMA, The Lancet, and The New England Journal of Medicine (NEJM) During the First Half of 1994 and 1998*

The quality of reports of RCTs in all 4 journals included in this study improved over time. This improvement appears to be greater for the journals that adopted CONSORT. However, because of our study design, it is only possible to suggest that the improvement may be associated with the implementation of the CONSORT statement.

These results also suggest that aspects of reporting of RCTs still require improvement. For example, the reporting of bias reduction methods, such as masking, is less than optimal. Similarly, our results confirm a concern raised by others10,11 regarding how the discussion/comment sections of RCTs are reported. It is unclear whether these deficiencies reflect difficulties in using CONSORT experienced by authors, by journals, or by both. To help address these questions, it will be important to obtain data from editors and authors as well as data on the readability of CONSORT reports as a way to gauge their scientific content.

We used 1 journal with a high citation impact factor as the comparator. This approach offers some control over more obvious forms of bias, such as the passage of time, and is considered to be a stronger research design than having no comparator.12 Ideally, we would have liked to include more comparator journals, but we were unable to identify them. Because of this, we limited our analysis in all cases except 1 to a comparison over time rather than between adopter and nonadopter journals. To strengthen these findings, we recommend that this evaluation be replicated and expanded to include more nonadopter journals. To facilitate such a study, we encourage all journals to indicate, perhaps in their information for authors, whether they support the CONSORT statement. Moreover, to increase generalizability of these results, future evaluations should also include specialty journals.13

Another limitation of our study is the time frame in which we completed the evaluation. We chose reports of RCTs published during the first half of 1998, only 12 to 18 months after the endorsement of CONSORT by journals included in this evaluation. It is possible, even likely, that effective dissemination is a slow process and that to estimate the true influence of CONSORT requires more time. In addition, our results pertain to the CONSORT checklist and do not evaluate the use of the CONSORT flow diagram. There are limitations to the present version of the flow diagram14 in terms of information requested of authors. Additionally, interpretation of the terms used and the sensibility of some of the criteria across RCTs has been inconsistent. Egger and colleagues15 report the results of an evaluation of the CONSORT statement flow diagram in an accompanying article.

In summary, these findings suggest that use of the CONSORT checklist may be associated with improving the quality of reports of RCTs. Higher-quality reports are likely to improve RCT interpretation, minimize biased conclusions, and ultimately facilitate decision making about treatment effectiveness.

Moher D, Pham B, Jones A.  et al.  Does quality of reports of randomized trials affect estimates of intervention efficacy reported in meta-analyses?  Lancet.1998;352:609-613.
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials.  JAMA.1995;273:408-412.
Begg CB, Cho MK, Eastwood S.  et al.  Improving the quality of reporting of randomized controlled trials: the CONSORT statement.  JAMA.1996;276:637-639.
Rennie D. How to report randomized controlled trials: the CONSORT statement.  JAMA.1996;276:649.
Altman DG. Better reporting of randomized controlled trials: the CONSORT statement.  BMJ.1996;313:570-571.
McNamee D, Horton R. Lies, damn lies, and reports of RCTs.  Lancet.1996;348:562.
Lebeau DL, Steinmann WC, Patel KV. Has the Randomized Controlled Trial Literature Improved After CONSORT? 50 Years of Clinical Trials: Past, Present and Future. London, England: BMJ Publishing Group; 1998.
Berlin JA.for the University of Pennsylvania Meta-analysis Blinding Study Group.  Does blinding of readers affect the results of meta-analyses?  Lancet.1997;350:185-186.
Jadad AR, Moore RA, Carroll D.  et al.  Assessing the quality of reports of randomized clinical trials: is blinding necessary?  Control Clin Trials.1996;17:1-12.
Clarke MJ, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents?  JAMA.1998;280:280-282.
Docherty M, Smith R. The case for structuring the discussion of scientific papers.  BMJ.1999;318:1224-1225.
Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for ResearchBoston, Mass: Houghton Mifflin Co; 1966:13-23.
Prusakowski N, Woo C, Hawkins BS. CONSORT and its effect on reports of clinical trials in ophthalmology journals [abstract].  Control Clin Trials.2000;21:65S.
Meinert CL. Beyond CONSORT: need for improved reporting standards for clinical trials.  JAMA.1998;279:1487-1489.
Egger M, Jüni P, Bartlett C.for the CONSORT group.  Value of flow diagrams in reports of randomized controlled trials.  JAMA.2001;285:1996-1999.

Figures

Tables

Table Graphic Jump LocationTable 1. CONSORT Checklist Criteria Included in Reports of Randomized Trials for Articles Published in BMJ, JAMA, The Lancet, and The New England Journal of Medicine (NEJM) During the First Half of 1994 and 1998*
Table Graphic Jump LocationTable 2. Quality of Reports of Randomized Trials, Using an Assessment Tool, for Articles Published in BMJ, JAMA, The Lancet, and The New England Journal of Medicine (NEJM) During the First Half of 1994 and 1998*

References

Moher D, Pham B, Jones A.  et al.  Does quality of reports of randomized trials affect estimates of intervention efficacy reported in meta-analyses?  Lancet.1998;352:609-613.
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials.  JAMA.1995;273:408-412.
Begg CB, Cho MK, Eastwood S.  et al.  Improving the quality of reporting of randomized controlled trials: the CONSORT statement.  JAMA.1996;276:637-639.
Rennie D. How to report randomized controlled trials: the CONSORT statement.  JAMA.1996;276:649.
Altman DG. Better reporting of randomized controlled trials: the CONSORT statement.  BMJ.1996;313:570-571.
McNamee D, Horton R. Lies, damn lies, and reports of RCTs.  Lancet.1996;348:562.
Lebeau DL, Steinmann WC, Patel KV. Has the Randomized Controlled Trial Literature Improved After CONSORT? 50 Years of Clinical Trials: Past, Present and Future. London, England: BMJ Publishing Group; 1998.
Berlin JA.for the University of Pennsylvania Meta-analysis Blinding Study Group.  Does blinding of readers affect the results of meta-analyses?  Lancet.1997;350:185-186.
Jadad AR, Moore RA, Carroll D.  et al.  Assessing the quality of reports of randomized clinical trials: is blinding necessary?  Control Clin Trials.1996;17:1-12.
Clarke MJ, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents?  JAMA.1998;280:280-282.
Docherty M, Smith R. The case for structuring the discussion of scientific papers.  BMJ.1999;318:1224-1225.
Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for ResearchBoston, Mass: Houghton Mifflin Co; 1966:13-23.
Prusakowski N, Woo C, Hawkins BS. CONSORT and its effect on reports of clinical trials in ophthalmology journals [abstract].  Control Clin Trials.2000;21:65S.
Meinert CL. Beyond CONSORT: need for improved reporting standards for clinical trials.  JAMA.1998;279:1487-1489.
Egger M, Jüni P, Bartlett C.for the CONSORT group.  Value of flow diagrams in reports of randomized controlled trials.  JAMA.2001;285:1996-1999.

Letters

CME
Meets CME requirements for:
Browse CME for all U.S. States
Accreditation Information
The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians. The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.
Note: You must get at least of the answers correct to pass this quiz.
You have not filled in all the answers to complete this quiz
The following questions were not answered:
Sorry, you have unsuccessfully completed this CME quiz with a score of
The following questions were not answered correctly:
Commitment to Change (optional):
Indicate what change(s) you will implement in your practice, if any, based on this CME course.
Your quiz results:
The filled radio buttons indicate your responses. The preferred responses are highlighted
For CME Course: A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes
Indicate what changes(s) you will implement in your practice, if any, based on this CME course.

Multimedia

Some tools below are only available to our subscribers or users with an online account.

Web of Science® Times Cited: 496

Related Content

Customize your page view by dragging & repositioning the boxes below.

See Also...
Articles Related By Topic
Related Collections