Reporting of Randomized Clinical Trial Descriptors and Use of Structured Abstracts FREE

STRUCTURED ABSTRACTS, that is, abstracts that describe a study using specified content headings rather that paragraph format, were suggested by the Ad Hoc Working Group for Critical Appraisal of the Medical Literature in 1987¹ "to provide more information . . . for articles reporting original research . . . of medical care."² The original objectives for using structured abstracts were to help health professionals quickly assess the reliability and content of a clinical report, to facilitate peer review, and to aid accurate indexing and retrieval of reports from computerized databases such as MEDLINE and EMBASE. In structured abstracts essential elements are concisely reported using content headings (eg, objective, design, setting, participants, intervention, results, conclusions).¹ Rapid adoption of structured abstracts by journals resulted in an annual doubling of reports with structured abstracts published from 1989 through 1991 appearing in MEDLINE; 15% of these were reports of clinical trials.³

Froom and Froom⁴ evaluated the quality of structured abstracts and found important deficiencies in reporting of patient demographics, follow-up, and statistical methods. Taddio et al⁵ compared structured with unstructured abstracts and found an improvement in reporting using the structured format, albeit with similar deficiencies. They could not rule out that improvements were due to changes over time. Harcourt et al³ reported that indexers at the National Library of Medicine applied an average of 3 more Medical Subject Headings to reports with structured abstracts compared with concurrent unstructured abstracts, but could not determine whether this finding was related to better abstract or text reporting or to differential indexing of the journals that first incorporated structured abstracts.

To our knowledge, no investigator has directly examined whether structured abstract use affects the reporting quality of report text. In writing about structured abstract merits, Rennie and Glass⁶ raise the possibility that "[abstract] structure reminds authors . . . of the necessity of providing each category of information." By focusing on the content headings required by a structured abstract, authors or editors might incorporate descriptors more consistently in the manuscript text. Our objective was to assess whether the use of structured abstracts is associated with an improvement in the overall reporting of ophthalmology randomized controlled trials (RCTs).

We selected 2 US ophthalmology journals that we previously found to regularly report large numbers of RCTs⁷ and that recently revised their Instructions to Authors to require structured abstracts, the Archives of Ophthalmology and Ophthalmology.^8- 9 We manually searched for reports describing RCTs the year preceding, of, and following the first appearance of structured abstracts (1992, 1993, and 1994, respectively, in the Archives of Ophthalmology and 1991, 1992, and 1993, respectively, in Ophthalmology). To monitor changes over time, we searched the American Journal of Ophthalmology (a US ophthalmology journal that publishes similar numbers of RCTs⁷ and that did not require structured abstracts until late 1994)¹⁰ for RCT reports from 1991 through 1994.

Hand searching was carried out by 2 independent, trained readers who examined each full-length report for RCT status in the selected journals for the specified years.¹¹ We defined RCT as a controlled experiment designed to evaluate an intervention or diagnostic tool, using a random method to assign individuals, eyes, or some other unit to a test or comparison group. We included quasi-randomized clinical trials, ie, those employing a method of assignment (eg, alternation) designed to avoid bias. Reports of RCTs that did not include data by randomized treatment group (eg, validation of a method used for measuring an outcome) were excluded.

We found 154 reports of RCTs. Five papers were excluded; 1 had no abstract, 2 presented data on subsets of patients, and 2 examined methods to measure outcomes. We also excluded 24 reports because the authors stated that an abbreviated methodological description was provided since the methods had previously been described.

Extraction of RCT Design and Operational Characteristics. The Consolidated Standards of Reporting Trials (CONSORT) statement describes RCT descriptors and provides a flowchart showing patient entry and follow-up that are recommended for inclusion in every RCT report.¹² Using CONSORT descriptors as a "gold standard" to evaluate reporting quality in each article, we scored the presence of 56 criteria (each flowchart block or descriptor, Table 1) in individual reports as yes, no, or not applicable. We selected 9 criteria (with daggers in Table 1) corresponding to common abstract content headings to measure inclusion in each abstract.

Because some study- or journal-specific characteristics could influence the inclusion of CONSORT statement descriptors, we extracted information by journal about study and report characteristics, including purpose of intervention, multicenter status, type of test intervention, sample size, group or individual authorship, length of report (number of pages), and length of methods section (number of pages).

Analyses. Data were entered into a database (Paradox, Version 4.0, Borland International Inc, Scotts Valley, Calif), and exported to a statistical program (SAS, Version 6.2, SAS Institute Inc, Cary, NC). We calculated the number of criteria included in each report, the proportion of reports that included specific criteria, and the number of reports that included more than the majority of criteria within each CONSORT subheading (introduction, protocol, etc). We compared structured with unstructured abstracts, and reports published in 1991 and 1992 with those published in 1993 and 1994 (American Journal of Ophthalmology), using the Student t test or χ² tests. Odds ratios (ORs) with 95% confidence intervals (CIs) are presented.

We found that RCTs reported in the Archives of Ophthalmology, compared with Ophthalmology and the American Journal of Ophthalmology, were more often multicentered (10 [38%] of 26 vs 12 [24%] of 51 and 13 [27%] of 48, respectively; P=.07), evaluated surgical or laser trials more often (7 [27%] of 26 vs 6 [12%] of 51 and 3 [6%] of 48, respectively; P=.001), more frequently devoted more than a single page to the methods section (10 [38%] of 26 vs 9 [18%] of 51 and 12 [25%] of 48, respectively; P=.005), and published more reports with study group authorship (5 [19%] of 26 vs 2 [4%] of 51 and 3 [6%] of 48, respectively; P=.001). Thus, we examined all results separately by journal.

Reporting of Descriptors in Text by Journal. Reporting of CONSORT criteria in the text was unimpressive. The mean (SEM) number of criteria included was 15.8 (0.4) of a possible 56; there was little difference among journals (Table 1). Journals were also remarkably similar in the proportion of reports that included specific criteria (Figure 1). Criteria reported in a low proportion of reports in all 3 journals were often associated with CONSORT subheadings associated with RCT methods, such as assignment and masking.

Comparison by Structured Abstract Use or Over Time. We found no difference in the mean number of criteria that were included in reports with structured abstracts compared with those without (Table 2). Including a majority of criteria within a single CONSORT subheading was positively associated with structured abstracts for "protocol" in the Archives of Ophthalmology (OR, 2.16; 95% CI, 1.12-4.16), negatively associated for "introduction" in Ophthalmology (OR, 0.57; 95% CI, 0.33-0.95), and not associated with later year of publication. Thus, we found no evidence for improvement in inclusion of criteria associated with structured abstracts.

We then calculated the proportion of reports in which a specific criterion was included to see if there were improved reporting by criterion associated with structured abstract use or year of publication. Individual criteria were included more or less often following structured abstract use or with later publication years, but by individual journal (Table 3).

Reporting of Descriptors in Abstract of Report. Of the 9 criteria used to evaluate abstract reporting, a mean of 5.0 (0.2) were included in all abstracts from all journals. Structured abstracts were more often associated with inclusion of criteria in the Archives of Ophthalmology (Table 2).

Specific criteria included infrequently in structured abstracts for the Archives of Ophthalmology, Ophthalmology, and American Journal of Ophthalmology were description study population (6 [67%] of 9, 18 [57%] of 28, and 16 [64%] of 25, respectively); primary outcome (6 [67%] of 9, 8 [29%] of 28, and 12 [48%] of 25, respectively); and number of patients followed up (2 [22%] of 9, 5 [18%] of 28, and 3 [12%] of 25, respectively).

Our results do not support an association of improved text reporting with structured abstracts or later publication in the journals examined. These 3 journals tended to be more alike than different in overall reporting of individual criteria, and there was no consistent pattern of change in reporting associated with structured abstract use or later publication. Possibly, the time period we examined was transitional or represented a lag time while editors or authors were incorporating use of structured abstracts. Haynes et al² reported that initial use of structured abstracts found some authors writing abstracts concurrently with manuscript preparation, whereas others did so only at submission or on editor request. Also, no special emphasis was placed on use of structured abstracts for RCTs initially, even with recognized importance for trial reports. Finally, it is possible that our sample size was insufficient to detect subtle changes in reporting, as our results are based on 125 reports from 3 ophthalmology journals; they may not be generalizable to other journals or areas of medicine.

Although we did not detect improvement in overall text reporting, we thought there might be individual criteria reported more frequently when structured abstracts were used, but found no consistent reporting pattern. For example, use of random or trial in the title and rationale for statistical tests were reported more often, but in separate journals. Some criteria were reported less frequently with structured abstracts, and perhaps were viewed as less important when space constraints limited text length. However, any changes we report in inclusion of criteria associated with structured abstract use or later year of publication may be due to chance, given the number of observations.

Although checklists for assessing reporting quality of RCTs were available,^13- 15 we chose to use CONSORT descriptors as a "gold standard" since it comprises a comprehensive list of criteria. We did not intend to evaluate trial quality. It has been argued, however, that "a well-designed but poorly reported trial could be judged as having low quality,"¹⁶ so assessing reporting is an important first step in assessing trial quality.

Our initial search yielded 24 RCT reports that included abbreviated methodological descriptions because methods had previously been reported. Editors are faced with a tension between space limitations and inclusion of all CONSORT descriptors in subsequent RCT reports. Since readers may not have previous reports available, we believe each RCT report should include all CONSORT descriptors to allow independent report evaluation.

Finally, we found a significant improvement in abstract reporting quality when structured abstracts were used in the Archives of Ophthalmology, and some improvement in Ophthalmology. Consistent with findings of others,^4- 5 we found abstract reporting deficiencies with authors frequently omitting a description of study population, primary outcome, or number of patients followed up.

In summary, we found no improvement in text reporting when structured abstracts were used. Nevertheless, structured abstract use should not be abandoned since abstract reporting itself is improved using this format. Rather, our results highlight the need for a standard such as the CONSORT statement to enhance RCT text reporting.