Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials FREE

In 2005, the International Committee of Medical Journal Editors (ICMJE) initiated a policy requiring investigators to deposit information about their randomized controlled trial (RCT) design into a clinical trial registry before participant enrollment as a precondition for publication of the trial's findings in member journals.¹ This policy required that principal investigators of all clinical trials starting enrollment after July 1, 2005, to register information about the study prior to participant enrollment. Registration usually involves reporting information on the 20 items proposed by the World Health Organization registration advisory group in April 2004.² The research community has embraced this policy, as seen by a marked increase in trial registration around the time of the implementation of the ICMJE policy.³ This great idea of trial registration switched rapidly from being ignored to being irresistible and trial registration that was some years ago the exception is now the rule.⁴

ClinicalTrials.gov received 30 new trial registrations per week, on average, before the implementation of the policy and 220 new trials per week, on average, since then.⁵

One of the main objectives of trial registration is to help achieve transparency in results and to make information about the existence and design of clinical trials publically available.^6- 14 This policy should permit knowledge sharing about the key elements of clinical trials and help decrease the risk of selective reporting of outcomes that was previously identified in published results of RCTs.^15- 17

Our study had 4 objectives: to assess the proportion of registered trials in a sample of RCTs whose results were recently published in journals with high impact factors; to compare in registered trials the primary outcomes registered with those reported in subsequent publications; to determine whether outcome reporting bias favored significant primary outcomes; and to compare general medical journals and specialty journals in terms of the proportion of registered trials and differences in primary outcomes.

We searched MEDLINE via PubMed to identify all reports of RCTs assessing treatments in 3 medical specialties (cardiology, rheumatology, and gastroenterology) indexed during 2008. This search was performed at least 3 months after the publication of articles (ie, up until March 2009). We used the limits of study type (“randomized controlled trials”) and journal type: the 10 journals indexed with the highest impact factors among the general and internal medicine category and the 10 journals indexed with the highest impact factors among journals in each of the 3 specialties, according to the Institute for Scientific Information Web of Knowledge (2007). The list of journals is in the Box. We focused on these 3 medical areas because of their important representation in clinical practice and clinical research. We decided to select only reports published in journals with the highest impact factor because they are all peer-reviewed and tend to be of higher quality.¹⁸

Journals Requiring Trial Registration for Publication, No. of Trials Included in Study

General Medicine

Cardiology

Rheumatology

Gastroenterology

Journals With No Information About Registration, No. of Trials Included

General Medicine

Cardiology

Rheumatology

Gastroenterology

One of us (S.M.) systematically examined the 2008 “Instructions for Authors” of the selected journals and checked whether they clearly requested the registration of the trial before submission of an article and referred to ICMJE guidelines about trial registration or to trial registration in general (the Box).

The title, keywords, and abstract of retrieved citations were assessed by 1 of us (S.M.). Articles were included if the study was identified as an RCT, which was defined a comparative study reporting a random assignment of participants (reporting “random,” “randomized” etc) anywhere in the report. We excluded commentary or discussion articles, observational studies (cohort studies, case-control studies, and cross-sectional studies), meta-analyses, ongoing studies, articles for which only the abstract was available, follow-up studies, and duplicate publications. After reading the full-text article, studies with no primary outcome explicitly identified anywhere in the text were also excluded.

One reviewer (S.M.), who was not blinded to authors and their affiliations, journal names, or funding sources, extracted all data using a standardized data extraction form. The following data were extracted.

The journal name and its 2007 impact factor were extracted. Data on the funding source of each published study was collected and categorized as no funding, government, industry, private nonprofit, other source, or not reported.¹⁹

From each published report, we reviewed the number and nature of the reported outcomes. An outcome was defined as a variable that was intended to be compared between randomized groups to assess the efficacy or harm of an intervention.¹⁵ Primary outcomes were those that were explicitly reported as such in the published article, along with the time frame of assessment. If none was explicitly reported, we used the outcome stated in the sample size estimation. If none was explicitly identified in the text or in the sample size calculation, the article was excluded. If information was available, we compared the primary outcomes explicitly reported as such in the published article with that stated in the sample size calculation.

For each published trial report, 1 of us (S.M.) systematically searched whether the trial was registered. We checked whether the authors reported registering their trial and whether the registration number was included in the published article.

When no registration number was reported in the published article, the corresponding author was contacted by e-mail to ask whether the trial was registered, and if so, in which registry, and the associated registration number. An initial e-mail was followed by 2 others a week apart when there was no response, initially. If we received no answer from the corresponding author, we searched the following clinical trial registries: ClinicalTrials.gov (NCT),²⁰ International Standard Randomized Controlled Trial Number Register (ISRCTN),²¹ and the registry of the country of the first or corresponding author (for example Australian New Zealand Clinical Trial Registry [ACTR] or Nederlands Trial Register [NTR]). We also searched the World Health Organization registry search portal.²²

To identify the record of these trials in the registries, we proceeded as follows:

If no registration number was found at the end of this process, the published article was considered not registered.

For each registered trial, we collected the dates reported as being the registration date and the start and end dates of participant enrollment. The ICMJE, in 2005, defined studies starting before July 1, 2005, as having to be registered before September 13, 2005, and studies starting after July 1, 2005, as having to be registered before the onset of participant enrollment. We analyzed the date of registration for both times separately. Finally, we excluded studies registered after the end of the study because it was not possible to evaluate the risk of outcome reporting bias in such studies.

We reviewed the number and nature of reported outcomes. An outcome was defined as a variable that was intended for comparison of randomized groups to assess the efficacy or harm of an intervention.¹⁵ Primary outcomes consisted of outcomes explicitly reported as such in the trial registration, along with the time frame of assessment. If none was explicitly identified, the trial's primary outcome was considered not registered.

For those explicitly identified, we checked whether it was reported with a clear description (ie, primary outcome reported with time-frame assessment). For example, “blood pressure” is not a clearly specified outcome. Ideally, we sought an unambiguous definition (eg, change in systolic pressure from baseline at 12 months). To take into account the amendments and possible changes by the data provider that could occur any time after the initial registration, when feasible, we checked all changes in the protocol that were available using a specific function (eg, “History of Changes” in ClinicalTrials.gov archive site).

For reports included in this study, we evaluated the consistency between the primary outcome(s) reported in the registry and those in the published article. For this analysis, we selected only trials registered before the end of the study, which gave a clear description of the primary outcome in the registry. For the end of the study, we used the date reported in the registry. If the term currently recruiting was in the registry, the study was excluded. If the terms ongoing, or not recruiting were in the registry, we noted the date reported in the part “end of the study” and analyzed the study according to this date. If the study was registered after the date of the end of the study reported in registry, we considered that the study was registered after the study completion. We collected information on the primary outcome reported in the article and that reported in the registry. We defined a discrepancy between the registered and published primary outcomes as follows: if the 2 were clearly different (eg, blood pressure and death) or were evaluated at a different times (eg, pain at 3 months and pain at 6 months).

If the 2 primary outcomes were different because the one reported in the registry was more imprecise than that reported in the article, we classified the study as having “imprecise outcome registration.” For example, in a study assessing rheumatoid arthritis, the registered primary outcomes were “clinical disease activity and radiographic change,” but the published ones were “DAS28-ERS<2.6 and the modified total Sharp score.”

Each primary outcome that was considered imprecise was confirmed by 2 of us (I.B., P.R.), with disagreements resolved by consensus.

We defined major discrepancies according to a modified classification of Chan et al¹⁵:

If the registry of the published text contained several primary outcomes, we applied the previous definition for each primary outcome. Discrepancies were identified by 1 of us (S.M.) and were confirmed by 2 other members (I.B., P.R.), with disagreements resolved by consensus.

We extracted the P values from the full-text article for all the primary outcomes that were registered and for all the outcomes reported in the article. We quoted results according to statistical significance: results significantly supporting or refuting the study intervention or favoring 1 of the groups in trials with 3 or more groups (ie, P < .05), results that did not reach statistical significance (ie, P ≥ .05), or unclear results. The same classification of results—significant, not significant, and unclear—for equivalence or noninferiority trials was according to the margin of equivalence set.

A discrepancy was considered to favor statistically significant results when a new statistically significant efficacy primary outcome was introduced or when a nonsignificant one was omitted or defined as nonprimary in the published article, according to the Chan classification.¹⁵ We also judged the discrepancy as positive when a new, statistically nonsignificant safety primary outcome was introduced in the published article. In fact, a new, statistically nonsignificant safety primary outcome was considered if the experimental drug had no more adverse effects than the comparator even though the trial was not powered to show a difference. The influence of some discrepancies could not be assessed because the published text contained no results concerning the registered primary outcome. Consequently, the positivity or negativity of the discrepancy was considered impossible to conclude, so the study was considered not assessable.

Median and interquartile ranges (IQRs) for quantitative variables and number of articles (percentages) for categorical variables were calculated for each study characteristic. Proportions of registered articles or differences in primary outcomes between general medical journals and specialty journals were compared by the χ² test or Fisher exact test, if necessary. Comparisons between general medical journals and specialty journals for quantitative variables involved the t test. P < .05 (2-tailed) was considered statistically significant. Statistical analysis involved use of SAS version 9.1 (SAS Institute Inc, Cary, North Carolina).

Of the 829 references retrieved, 413 reports were selected after screening titles, abstracts, and keywords (Figure). Five articles reported the results of 2 RCTs, and for these, we focused only on the first RCT whose results were reported.

The description of included articles is presented in Table 1. Of the 323 retained articles, 114 (35.3%) were published in general medical journals and 209 (64.7%) in specialty journals. Fifteen specialty journals (ie, 48 articles; 14.9%) provided no guidance on trial registration in their instructions to authors.

The clinical content for 44.0% of the articles was cardiology; 31.6%, gastroenterology; and 24.5%, rheumatology. An industry funding source was reported for 182 articles (56.3%), 76 (66.7%) of the 114 articles published in general medical journals, and 106 (50.7%) of the 209 in specialty journals.

Sixty-six articles (20.4%) provided no description of the sample size calculation for the trial. When the sample size calculation was reported, the primary outcome identified in the text and that used for sample size calculation was the same for 243 articles (94.6%) but differed for 14 articles (5.4%).

A total of 147 trials (45.5%) were adequately registered (ie, registered before the end of the trial, with the primary outcome clearly specified). Trial registration was lacking for 89 published reports (27.6%), 45 trials (13.9%) were registered after the completion of the study, 39 (12.1%) were registered with no or an unclear description of the primary outcome, and 3 (0.9%) were registered after the completion of the study and had an unclear description of the primary outcome.

The proportion of registered trials was greater for articles in general medical journals than for those in specialty journals (111 of 114 [97.4%] vs 123 of 209 [58.9%]; Figure). All but 3 articles published in a general medical journals reported the registration number. Among specialty journal articles, 94 trials (76.4%) reported the registration number, 23 (18.7%) were provided by the corresponding author, and 6 (4.9%) were found after searching clinical trial registries for trials.

Most of the trial protocols were registered in either ClinicalTrials.gov (197 of 234 [84.2%]) or ISRCTN (28 of 234 [12.0%]).

Of the 323 articles assessed, 48 (6.7%) were published in a journal containing no information about trial registration in the instructions for authors, and 32 of these trials were registered. Sixty-nine trials published in journals that mentioned registration in their instructions for authors were not registered.

Of the 234 trials that were registered, 87 (37.2%) were not taken into account because they were registered after the completion of the study (n = 45), because of imprecise or missing description of the primary outcome or the timing of its assessment in the registry (n = 39), or both (n = 3; Figure).

Among the 147 remaining reports, the primary outcomes of 46 articles (31.3%) differed from that registered (Table 2); this proportion was similar for general medical journals (22 of 75 [29.3%]) and specialty journals (24 of 72 [33.3%]; P = .73). The discrepancies consisted of the introduction of a new primary outcome in the article (ie, a secondary outcome or an absent outcome in the registry that becomes a primary outcome; 22 of 147 [15.0%]), omission of the registered primary outcome from the article (15 of 147 [10.2%]), published primary outcome registered as a secondary outcome (8 of 147 [5.4%]), a registered primary outcome reported as a secondary outcome in the article (6 of 147 [4.0%]), and timing of assessment different in the article and the registry (4 of 147 [2.7%]). Nine articles had 2 reasons for differences in primary outcomes.

For the 46 articles with a discrepancy between the registry and the published article, the influence of this discrepancy could be assessed only in half (23 of 46). Among them, 19 of 23 (82.6%) had a discrepancy that favored statistically significant results (ie, a new, statistically significant primary outcome was introduced in the published article or a nonsignificant primary outcome was omitted or not defined as the primary outcome in the published article).

For the 147 trials registered before the end of the participant recruitment, 101 had a clear description of the primary outcome in the registry that matched the published primary outcome; the proportion was not different between general medical and specialty journals. These trials represent less than half of the 234 registered trials (101 of 234; 43.2%).

To evaluate trial registration and selective outcome reporting bias, we identified a sample of 323 recently published RCTs in cardiology, gastroenterology, and rheumatology. Most journals provided guidance on trial registration in their instruction to authors section. Only 147 trials (45.5%) were adequately registered (ie, registered before the end of the trial, with the primary outcome clearly identified). Registration was missing for 90 trials (27.9%). Forty-five trials (13.9%) were registered after the completion of the study, 39 (12%) were registered with no or an unclear description of the primary outcome, and 3 (0.9%) were registered after the completion of the study with no or an unclear description of the primary outcome. Furthermore, 31.3% (46 of 147) of trials adequately registered showed some evidence of selective outcome reporting. Of these 46 articles, when the primary outcome description was available, 82.6% of differences in primary outcomes between the registration and article (19 of 23) favored reporting statistically significant results, with the remainder unclear. Finally, when comparing general medical journals and specialty journals, results of trials published in general medical journals were more often adequately registered, but selective outcome reporting did not differ between the 2 types of journals.

A main goal of trial registration is to enhance transparency of research and accountability in the planning, conduct, and reporting of clinical trials, an objective achieved by making available details about the trial.^1,23- 24 Therefore, adequate registration should be a safeguard against publication bias. A major step has been achieved with the ICJME initiative for trial registration, and the existence of all trials is now publicly available. However, after this first step, the quality and timing of registration still needs improvement. Consistent with our results, Reveiz et al²⁵ found that among the 275 respondents to a survey sent to a random sample of corresponding authors of published clinical trials, only 21% of them had registered all of their ongoing trials since 2005, but 47% stated that they would provide the 20 World Health Organization data set items to a registry for all their future clinical trials.²⁶ In 2005, Zarin et al²⁷ found inconsistent quality of information for trials registered in ClinicalTrials.gov. Sekeres et al²⁸ reported frequent omission of scientific leadership (name of the study chair or principal investigator; 355 of 1388 [26%] and contact e-mail addresses, 855 of 1388 [62%]) in trial registration.

Adequate trial registration could also be a great opportunity to evaluate and limit other biases such as outcome reporting bias. However, our results highlight that editors and peer reviewers did not take advantage of the transparency provided by registration. Although they had the opportunity to check outcome reporting bias by comparing submitted manuscripts to information available for the trial registration, we found some evidence of selective outcome reporting in 28% of the articles adequately registered.

Previously, Chan et al¹⁶ reported a difference in primary outcomes between trial protocols and publications for 40% of the trials the authors analyzed. In another study, these authors found 62% of trials with at least 1 primary outcome that was changed, introduced, or omitted from that in the submitted protocol¹⁵; among the 51 trials with major discrepancies, 16 (31.4%) had discrepancies that favored statistically significant primary outcomes in the articles. In a recent survey of protocols and subsequent publications from the Lancet, major differences were observed for the primary outcome in 11 of 37 trials.²⁹ The results of these different studies performed with specific samples of trials (trials approved from 1990 to 1998 in Canada and between 1994 and 1995 in Denmark or trials published in a specific journal), and a different method could not be directly compared with our results, which were obtained in a large sample of recently published articles from various journals.

These results highlight several issues. First, it is important for the entire scientific community to endorse and adhere to the requirement for trial registration before the beginning of participant enrollment in a trial. Although most trials with results published in general medical journals with high impact factors were registered, several did not respect the deadline of registration imposed by editors. Furthermore, less than half of the trials with results published in specialty journals were registered. The involvement of institutional review boards³⁰ and funders of trials³¹ has been advocated to help to enforce trial registration. Legislation making trial registration mandatory could also be a major incentive for trial registration. Despite many journals following the ICMJE guidelines and requiring a registration number for publication, some journals use vague language such as “We encourage the registration of all interventional trials” [emphasis added] in their instructions to authors. A lack of clarity in submission guidelines journal guidance might help explain inadequate trial registration. Unclear language has also been suggested for a lack of adherence to reporting guidelines.³²

Second, if the data in registries are not sufficiently detailed, with some sections not reported or inadequately reported, these data will not be useful. The sponsor and principal investigator are responsible for a trial's proper registration. Furthermore, quality control procedures of trial registries could be improved.

Trial registration provides a good opportunity for editors, peer-reviewers, and policy makers to identify outcome reporting bias and other deviations from the planned study to prevent such distortions from reaching publication. Outcome reporting bias is widely accepted as being a major problem that deserves more attention.^{15- 16,33- 35} Recently, the Cochrane collaboration has developed a tool to assess the overall risk of bias of trials included in systematic reviews. Selective outcome reporting is one essential component of this tool.³⁶ As highlighted in our results, which confirm earlier findings,¹⁶ changes to the outcomes reported mainly favored statistically significant results over statistically nonsignificant results. Such changes made after seeing all the results lead to biased and misleading results and are unacceptable. Possible reasons are to provide evidence that the treatment should be prescribed to patients or perhaps simply to increase the chance of acceptance of the report for publication, but the determinants of this bias need to be investigated further. Before the requirement of trial registration, evaluation of outcome reporting bias was difficult because it implied obtaining the trial protocol. Thanks to trial registration, identifying and quantifying outcome reporting bias as routine is feasible when appraising the quality of published results of an RCT.

This study has some limitations. First, we selected only trials published in journals with a high impact factor; for more than 90% of the selected reports, these journals endorsed trial registration in their instructions for authors. Consequently, these results probably underestimate the amount of inadequate and nonexistent trial registration. Second, this study was restricted to 3 medical specialties, which could cause difficulties of generalization. Third, of the 234 registered trials, 195 (83.3%) enrolled participants before the implementation date specified in the ICMJE policy regarding trial protocol registration (ie, July 1, 2005). Consequently, the results could improve for trials beginning after that date because researchers are more likely to be aware of this rule and editors could be stricter for such trials. However, of the 39 trials with the first participant enrolled after the July 1, 2005, deadline, only 5 were registered before the onset of the study and 4 more after the end of the study. Furthermore, some trials may not have been adequately registered or data were not updated in the registry after protocol amendments, and the registered primary outcome was different from that specified in the protocol submitted to the research ethics committee. Finally, we focused on only primary outcomes and did not address the problem of unreported secondary outcomes.

To obtain the full benefit of clinical trial registration, the active participation of different stakeholders is needed. First, the sponsor and principal investigator should ensure that the trial details are registered before enrolling participants. Second, the comprehensiveness of the registration should be routinely checked by editors and readers, especially regarding the adequate reporting of important items such as the primary outcome. Third, editors and, possibly, peer reviewers should systematically check the consistency between the registered protocol and the submitted manuscript to identify any discrepancies and, if necessary, require explanations from the authors. Finally, the goal of trial registration could be to make available and visible information about the existence and design of any trial and give full access to all trial protocols³⁷ and the main trial results.³

Recently, ClinicalTrials.gov has introduced the reporting of basic results associated with registered trials. When authors report results—mandatory in the United States for trials of drugs (including biologics) and devices subject to Food and Drug Administration regulation—it will be increasingly difficult to hide information.

In conclusion, although trial registration is now the rule, careful implementation of trial registration, with full involvement of authors, editors, and reviewers is necessary to ensure publication of quality, unbiased results.