Adherence to Published Standards of Reporting:  A Comparison of Placebo-Controlled Trials Published in English or German FREE

QUALITY of scientific work is judged at several stages from the initiation of research projects to the application of their results in clinical practice. Recent research has greatly contributed to improving quality through the process of peer review in publishing^1- 2 and the systematic appraisal of trial quality in systematic reviews.^3- 7 Within systematic reviews^8- 11 and studies assessing publication quality,^12- 17 various checklists and scales have been used. However, missing definitions of "trial quality" have been identified as a basic conceptual problem in most studies.^18- 19 In addition, in methodologic quality assessment, the quality of reporting should be distinguished from the methodologic quality.⁵

Defined standards in an accepted form are basic for quality assessment²⁰ and have existed as Uniform Requirements for Manuscripts Submitted to Biomedical Journals for 20 years.²¹ The Uniform Requirements are now accepted by over 500 journals.²² The objective of the present study was to improve reporting quality by measuring the reporting quality of articles on randomized controlled trials (RCTs) and to identify those aspects of trial reports with the largest deficiencies. At the same time, this study compared the reporting quality of articles written in English or German by the same group of authors.

All 224 trial reports published from 1985 through 1994 in 5 leading German-language general medical journals were identified in a manual search. Subsequently, 559 reports published in English by the same key authors (first, second, and last authors) during the same 10-year period were identified through MEDLINE. This sample of studies had been used previously to study aspects of language bias^23- 24 and of trial quality.²⁵

To create a homogeneous group of trials, all 131 reports of placebo-controlled trials with parallel design were selected from these 783 articles. Ten duplicates reporting the same study with the same outcomes were excluded, using a random procedure. This resulted in a final sample of 32 German and 89 English trial reports in 5 and 58 journals, respectively.

Outcome Measures. We defined report quality as "adherence to published standards on trial reporting." A scale for the assessment of reporting quality items was constructed from the checklist for statistical review of articles on clinical trials for BMJ²⁶ and from the third edition of the Uniform Requirements.²⁷ Items were interpreted according to references.^13,28- 29

We used 14 of the 24 items of the BMJ checklist and 11 of 12 aspects of the Uniform Requirements (Table 1). Items were not used when they were not applicable to the selected sample or when their assessment required clinical specialty knowledge. Three items in the BMJ checklist on blinding and completeness of follow-up reflected trial quality and could not be used in the scale of reporting quality. The 18 items used could be answered with a yes or no. Unclear situations were treated as not meeting the criteria. The quality score was calculated as the sum of met criteria. Data were analyzed with SAS for Windows 6.10, Proc Freq, and Proc Univariate (SAS Institute, Cary, NC).

The mean quality score was 8.4 (SD, 3.0; range 1-16). Fifty percent of the 121 reports met 9 or more of the 18 criteria (Table 1). Most frequently, information was missing in articles for confidence interval (96%), method of randomization (93%), reference of the statistical software used (90%), and sample size calculation (89%). However, a sufficient description of the experimental treatment was missing in only 7% of articles. Criteria were met more often for clinical aspects than for trial methodology and statistical analysis.

Differences by Language. The difference of the mean quality score between English- and German-language reports was 0.27 (95% confidence interval, −0.97 to 1.52). Seventeen of the items showed no difference between German-language and English-language reports (P>.1 in χ² or Fisher exact tests). Side effects were reported in 63% of German and 39% of English reports (P=.04 in χ² test).

Duplicate Publications. There were 11 sets with 2 to 4 papers in each set concerning the same study, altogether 26 articles reporting on 12 studies (in 1 case, 2 studies were reported in 3 papers). Previous publication was handled and referenced according to standard rules²² in only 4 cases. Four studies had been reported in identical or similar reports and in 1 case 4 papers had increasing numbers of patients. In another case, 2 papers reported on different subgroups of the same study without cross reference.

Three such sets were in English only, 1 in German only, and 7 with articles in both languages. In addition, 5 articles gave a reference to a previous publication or stated that they were preliminary reports, where the counterpart was not in the sample. Six English and 4 German papers were excluded from the analysis because they were identical with another paper.

Using a sample of reports from German-speaking Europe, the present study confirms the existence of large deficiencies in the reporting of placebo-controlled randomized trials with parallel designs. Missing sample size calculations and missing descriptions of the method of randomization are major deficits. These have been reported earlier.^8,13,15 The results suggest that it may be easier for investigators to describe clinical features (inclusion criteria and therapeutic regimens) than to report trial methodology and statistical features of their studies. Yet, incomplete reporting of trial methodology impedes reading and systematic reviewing of articles.

No difference between German and English articles was found. The size of the sample would have allowed the discovery of a difference in the scores of 2 with a power of .9 on a 2-sided test at the α level of .05.

There are a number of limitations to the present study. The concept of quality poses some problems. The basic assumption underlying the measurement of quality in a single score is that items of quality are correlated with each other. This can be questioned because 1 fatal error can invalidate the results of otherwise excellent work. However, we are often not aware of the particular excluding criteria. Also, only papers already published were investigated. Within the present study, scoring of report quality was not duplicated and interrater reliability could therefore not be evaluated. External validity was aimed at by using accepted standards. Finally, only articles of authors from German-speaking Europe were assessed. A further study is under way to measure the reliability of the scale used and to compare the present sample with articles from English-language countries.

Standardized assessment of papers submitted for publication can improve the adherence of authors to these standards.^14,16 Adherence could be promoted by better dissemination of content and meaning of the standards such as the Uniform Requirements. The Uniform Requirements are also known as "The Vancouver Style," suggesting that the Uniform Requirements only refer to the style of manuscripts, but not to methodological and statistical reporting. MEDLINE references 58 publications of the 5 editions of the Uniform Requirements in 8 languages. A German translation of the fourth edition was published in 1993, but only by an ophthalmology journal.³⁰

The Consolidated Standards of Reporting Trials (CONSORT) statement is an important step toward improving reporting quality of RCTs since it requires that authors themselves complete the checklist on reporting quality.³¹ It might be recommended that all journals, including those not following the CONSORT statement strictly, request their authors to fill out a checklist on reporting quality.