Use of the CONSORT Statement and Quality of Reports of Randomized Trials:  A Comparative Before-and-After Evaluation FREE

Reports of randomized controlled trials (RCTs) are the "gold standard" by which health care professionals and others make decisions about treatment effectiveness. To assess the strengths and limitations of RCTs, readers need and deserve to know the quality of their methods. Previous studies^1- 2 indicate that reports of low-quality RCTs, compared with reports of higher-quality ones, overestimate the effectiveness of interventions by about 30% across a variety of health care conditions.

The Consolidated Standards for Reporting of Trials (CONSORT) Group developed the CONSORT statement,³ an evidence-based approach to help improve the quality of reports of RCTs. Since its publication in 1996, the CONSORT statement has been widely supported,^4- 6 has been translated into several languages, and has an Internet presence (http://www.consort-statement.org) to facilitate awareness and dissemination. However, there is a paucity of data⁷ regarding whether the CONSORT statement has improved the quality of reports of RCTs.

We compared a sample of RCTs published in 1994 (pre-CONSORT) to a sample published in 1998 (post-CONSORT). To be included, RCT reports had to be drawn from journals classified by the Science Citation Index as "general and/or internal medicine." Four journals were evaluated: BMJ, JAMA, The Lancet, and The New England Journal of Medicine (NEJM). Three of these journals (BMJ, JAMA, Lancet) were early adopters of the CONSORT statement,^4- 6 whereas the fourth (NEJM) did not formally adopt the CONSORT statement and was used as a comparator.

Reports were included only if they involved human subjects and if the study design was identified as an RCT by examining the title and the abstract. Two readers (A.J., L.L.) independently hand-searched the 4 journals for RCTs published between January 1 and June 30 of both 1994 and 1998. Hard copies of relevant articles were obtained but were not masked because evidence concerning the effect of masking on assessments of trial quality is inconsistent.^8- 9

Three measures were used to assess the quality of reports of RCTs. First, the CONSORT checklist was modified so that multiple items were listed separately, which resulted in 40 items. Each item was assigned a yes or no response depending on whether the authors had reported it. Second, the reporting of allocation concealment was assessed as adequate, inadequate, or unclear.² Third, the Jadad scale,⁹ which contains 2 questions for randomization and masking and 1 question evaluating the reporting of withdrawals and dropouts, was used to assess quality. Each question entails a yes or no response option. In total, 5 points can be awarded, with higher scores indicating superior quality. Two reviewers (A.J., L.L.) completed all of these evaluations.

Both reviewers underwent training in evaluating RCTs using the CONSORT checklist. Before training, the definition of each checklist item was discussed. To assess interobserver agreement, 5 items from the checklist were purposefully selected (inclusion criteria, exclusion criteria, point estimate, deviation from protocol, and general interpretation of study findings). A κ statistic was calculated for each item based on a randomly selected set of 10 RCTs, from 1994 and 1998, and these were not included in this study. Discrepancies were resolved by consensus involving a third party (D.M.). A similar approach was used to assess interobserver agreement in assessing reporting of allocation concealment and using the Jadad scale.

Taking time (1994-1998) into consideration and using either unpaired t tests or χ² tests, we compared the number of checklist criteria included in each report and the mean number of criteria included within each subheading specified in the CONSORT checklist. We also assessed the percentage of studies that reported unclear allocation concealment and the specific item and overall quality score derived from the Jadad scale. The number of CONSORT checklist items reported was treated as the dependent variable in an analysis of variance (ANOVA) with 2 factors: journal and year. The difference between the adopter journals and the comparator journal was derived from the fitted ANOVA using a contrast involving journal (ie, adopters vs comparator) and year (ie, 1998 vs 1994). Least-square estimates and 95% confidence intervals (CIs) also were computed; for significant level, P = .05 (2-sided).

Of the 221 RCTs identified, 211 met the inclusion criteria. Six studies were excluded because they were animal studies and 4 studies because they were quasi-randomized trials.

Substantial agreement was established for 4 items from the CONSORT checklist and the Jadad instrument (inclusion and exclusion criteria, point estimate and general interpretation [κ = 1.0], quality assessment, and overall Jadad score [κ = 0.74]). Moderate agreement was established for allocation concealment (κ = 0.53) and for 1 item from the CONSORT checklist (κ = 0.54, deviation from protocol).

There was an increase over time in the number of CONSORT checklist items included in the reports of RCTs in all 4 journals (Table 1). This increase was statistically significant for 2 individual journals and overall for adopter journals (pre-CONSORT, 23.4; mean change, 3.7; 95% CI, 2.1-5.3). Over time, the increase in the reporting of CONSORT items was significantly greater for adopter journals when evaluated against the comparator journal (mean difference, 3.8; 95% CI, 1.0-6.5; 2-sided P = .007).

The proportion of RCTs with unclear reporting of allocation concealment decreased over time in all 4 journals (Table 2) and was statistically significant for adopter journals (pre-CONSORT, 61%; mean change, −22%; 95% CI, −38% to −6%). Over time, 3 of the 4 journals improved the quality of reports of RCTs as assessed by the Jadad scale (Table 2), which was statistically significant for 1 journal (Lancet) and across the adopter journals (pre-CONSORT, 2.7; mean change, 0.4; 95% CI, 0.1-0.8).

The quality of reports of RCTs in all 4 journals included in this study improved over time. This improvement appears to be greater for the journals that adopted CONSORT. However, because of our study design, it is only possible to suggest that the improvement may be associated with the implementation of the CONSORT statement.

These results also suggest that aspects of reporting of RCTs still require improvement. For example, the reporting of bias reduction methods, such as masking, is less than optimal. Similarly, our results confirm a concern raised by others^10- 11 regarding how the discussion/comment sections of RCTs are reported. It is unclear whether these deficiencies reflect difficulties in using CONSORT experienced by authors, by journals, or by both. To help address these questions, it will be important to obtain data from editors and authors as well as data on the readability of CONSORT reports as a way to gauge their scientific content.

We used 1 journal with a high citation impact factor as the comparator. This approach offers some control over more obvious forms of bias, such as the passage of time, and is considered to be a stronger research design than having no comparator.¹² Ideally, we would have liked to include more comparator journals, but we were unable to identify them. Because of this, we limited our analysis in all cases except 1 to a comparison over time rather than between adopter and nonadopter journals. To strengthen these findings, we recommend that this evaluation be replicated and expanded to include more nonadopter journals. To facilitate such a study, we encourage all journals to indicate, perhaps in their information for authors, whether they support the CONSORT statement. Moreover, to increase generalizability of these results, future evaluations should also include specialty journals.¹³

Another limitation of our study is the time frame in which we completed the evaluation. We chose reports of RCTs published during the first half of 1998, only 12 to 18 months after the endorsement of CONSORT by journals included in this evaluation. It is possible, even likely, that effective dissemination is a slow process and that to estimate the true influence of CONSORT requires more time. In addition, our results pertain to the CONSORT checklist and do not evaluate the use of the CONSORT flow diagram. There are limitations to the present version of the flow diagram¹⁴ in terms of information requested of authors. Additionally, interpretation of the terms used and the sensibility of some of the criteria across RCTs has been inconsistent. Egger and colleagues¹⁵ report the results of an evaluation of the CONSORT statement flow diagram in an accompanying article.

In summary, these findings suggest that use of the CONSORT checklist may be associated with improving the quality of reports of RCTs. Higher-quality reports are likely to improve RCT interpretation, minimize biased conclusions, and ultimately facilitate decision making about treatment effectiveness.