Transition From Meeting Abstract to Full-length Journal Article for Randomized Controlled Trials FREE

In the past decade, most major scientific meetings have developed mechanisms to highlight those research studies deemed to be of major clinical import by meeting organizers. For example, late breaking clinical trials sessions have been incorporated into 12 of the 17 annual meetings of US specialty societies. Presentations of randomized controlled trials (RCTs) presented at these late-breaking trials sessions are often featured promptly and prominently in the general media, on Web sites targeting health professionals, and in industry-sponsored mailings to practicing physicians who may not have attended the meeting.

One of the key questions facing clinicians and authors of meta-analyses or practice guidelines is whether data presented in abstracts at scientific meetings but not yet published as full-length peer-reviewed journal articles should be incorporated into the evidence base for practice. Proponents of accepting meeting abstracts as evidence argue that to do otherwise permits publication bias to influence practice (since less than half of all research presented at scientific meetings is ever published and studies with positive results are more likely to lead to full-length articles).¹ Thus, 78% of meta-analysis authors surveyed thought that abstracts should be included in summaries of the evidence.²

On the other hand, there is a potential risk in accepting meeting abstracts as evidence if the data presented are incomplete or if the interpretation of the data change after peer review.³ Thus, 53% of journal editors thought that meeting abstracts should not be included in evidence summaries.² Indeed, studies^4- 6 have demonstrated substantial differences between data presented in meeting abstracts and subsequent full-length journal articles at 2 surgical meetings and a general pediatric meeting, however, the majority of the abstracts in these 3 studies described nonrandomized observational studies.

To date, no studies have examined the impact of late-breaking trials sessions on publication rates nor consistency of data between data presented in abstracts and subsequent full-length journal articles. Thus, we undertook the current study to evaluate whether presentation in the late-breaking trials session of a major specialty society meeting compared with presentation elsewhere in the meeting affected: (1) the proportion of RCTs that are subsequently published as full-length journal articles; (2) the rapidity with which they are published; and (3) the consistency between the data presented in the meeting abstract and the subsequent full-length publication.

All RCTs presented at the late-breaking clinical trials session of the American College of Cardiology (ACC) scientific meetings between 1999 and 2002 were included if they reported clinical outcomes in human participants. We identified these RCTs from a search of the ACC Scientific Sessions Web site, theheart.org Web site (for the relevant meeting dates), and publications summarizing the late-breaking clinical trials session at each ACC meeting in the Journal of the American College of Cardiology . We chose to examine publication outcomes for RCT abstracts presented at least 3 years before the time of our literature search (ie, up to and including the March 2002 ACC meeting) to align with the recommendations arising from a Cochrane review that demonstrated that the median time for publication of RCTs was 18 months.¹

To generate a comparison group of RCTs, 2 investigators independently hand searched the 1999 to 2002 ACC Scientific Sessions abstract proceedings booklets to identify RCTs that were presented in oral or poster sessions but not in the late-breaking clinical trials session at each meeting—those RCTs that did not include human participants or reported only nonclinical outcomes (ie, cost) were excluded. Of the 8829 total abstracts published in the ACC proceedings booklets for these 4 years, 432 abstracts described RCTs that met our eligibility criteria.

In choosing our sample size for this study, we estimated that the publication rate for non–late-breaking RCTs would be 60% (based on the 58% rate demonstrated for RCTs in other topic areas),¹ and in order to detect (or rule out with 80% power) a 20% absolute increase, we thus needed at least 182 trials. After reviewing the 2000 ACC meeting late-breaking trials sessions, we anticipated an average of 24 late-breaking trials per ACC meeting and thus, a priori set a sample size of 4 years' worth of ACC late-breaking trials with a comparison group of 25 RCTs presented at regular sessions of the same ACC meeting each year. The comparison RCT abstracts were randomly selected and were not matched on any characterstics (beyond year of presentation) with the late-breaking trials abstracts.

To determine if abstracts had been subsequently published as full-length journal articles, we searched PubMed, MEDLINE, EMBASE, and the Cochrane Central Register of Controlled Trials databases for title, author(s), trial name and/or acronym, title keywords, and a combination of these terms (search last updated on December 8, 2005). If this search did not locate an article, then a secondary search was conducted using the same search strategy in SIGLE (System for Information on gray literature in Europe), Google, Cardiosource, and Incirculation (entering the complete abstract title using quotations to derive Web findings that contained those exact titles).

For an abstract to be considered published as a full-length journal article, the corresponding manuscript had to describe the same intervention and have either the same trial name, acronym, or at least 1 author who was named on the meeting abstract plus at least 1 outcome that was presented in the meeting abstract.

The data in Table 1 and Table 2 were extracted from each abstract and its corresponding full-length publication (if applicable).^7- 8 For the purposes of our study, the primary outcome for each RCT was defined as that presented in the ACC abstract.

Two investigators independently identified eligible RCT abstracts, conducted the literature searches for corresponding full-length articles, and extracted prespecified data, with each investigator cross-checking the other's data. Any discrepancies were resolved through consensus; if consensus could not be reached, a third investigator was consulted.

The t test and χ² test (or Fisher exact test when sample sizes were small) were used to evaluate differences in the binary and continuous variables in Table 1 between RCTs presented at the late-breaking clinical trials sessions and other sessions. The McNemar test and paired t test were used for the paired comparisons in Table 2 between RCTs that were vs were not subsequently published. Statistical significance was defined as P<.05. Publication times between late-breaking trials and other RCTs were compared using the log-rank test (unadjusted). A multivariable proportional hazards model was run to test which covariates affected publication time and to calculate the adjusted hazard ratios for late-breaking trials vs other RCTs. Variables used in the model were presentation session (late-breaking or other), trial sample size (logarithmic scale), trial conclusion (significant finding or not), RCT design (crossover/parallel), purpose of RCT (superiority/other), multicenter trial (yes/no), university-affiliated (yes/no), principal investigators affiliated with a university in the United States (yes/no), and source of funding (industry/not stated).

To determine whether point estimates for primary outcomes were different from abstract to publication, an effect size was calculated for each abstract and full-length article using available data (effect sizes, percentages, or proportions). For dichotomous data, this was calculated as a risk difference and for continuous data, it was calculated as a mean difference. Abstract/published article differences that were attributable to different rounding thresholds (ie, the value with the larger number of significant digits could be rounded to the one with fewer significant digits) were not counted as discrepancies. For those abstract/manuscript pairs that had different point estimates, P values and confidence intervals were calculated or directly used to determine whether there was a change in significance and confidence interval. A logistic regression was run on the probability of a difference in primary outcome effect estimates from meeting abstract to full-length article using time to publication as well as the variables listed in the Cox model as covariates. Whenever possible, we calculated a standardized effect size (in units of SD) for the difference between published article and abstract. For continuous measures, this was done by dividing the mean difference by the SD, while for dichotomous data the log risk ratio was calculated and divided by its corresponding SE. These values were used to determine the magnitude of the differences between abstract and published article. The between-groups effect sizes were compared using the Mann-Whitney test due to the skewed nature of the values and the unequal SDs.

Data analysis was performed using SAS statistical software version 8.02 (SAS Institute Inc, Cary, NC).

Of the 86 late-breaking clinical trials presented at the ACC between 1999 and 2002 (eTable 1), 79 (92%) were published prior to December 8, 2005 (with publication rates varying from 86% of the 21 late-breaking clinical trials presented in 2002 compared with 100% of the 18 late-breaking trials from 1999). In comparison, 69% of the clinical trials presented at other sessions of the ACC (eTable 2) were published prior to December 8, 2005 (with publication rates varying from 60% of those RCT abstracts presented in 2000 to 80% of those presented in 2002). All full-length publications were found in our primary literature search of PubMed, MEDLINE, and EMBASE. RCTs presented at the late-breaking clinical trials sessions were much more likely to be published (Figure 1) and were published much sooner than RCTs presented in other sessions of the ACC (median time to publication, 11.5 months vs 22 months, P<.001; Figure 2). In the multivariate proportional hazards model adjusting for those other factors (Table 1) independently associated with time to publication (sample size, conclusion of study, and RCT design), RCTs presented in the late-breaking sessions were still much more likely to be published and published sooner (adjusted hazard ratio, 1.80; 95% confidence interval, 1.24-2.61) than RCTs presented at other sessions of the ACC.

The 148 study abstracts that resulted in full-length publications were published in a total of 28 journals, with the most frequent destinations being Circulation (36 publications), Lancet (21 publications), The New England Journal of Medicine (16 publications), the Journal of the American College of Cardiology (15 publications), and JAMA (10 publications). Late-breaking trials were more likely to appear in these journals (the 5 journals publishing cardiology research with the highest impact factors) than trials presented at other sessions of the ACC (68 of 79 [86%] vs 30 of 69 [43%]; P<.001).

RCTs presented at the late-breaking sessions were significantly larger and were more likely to have principal investigators affiliated with US universities than RCTs presented at other ACC sessions (Table 1). Late-breaking RCTs were much less likely to report favorable results for the intervention than other RCTs, and were also more likely to have design papers published prior to the meeting (Table 1). The Jadad quality scores derived from review of their subsequent full-length articles were higher for those RCTs presented at the late-breaking sessions compared with those presented at other ACC sessions (mean [SD], 2.69 [1.06] vs 2.19 [1.30]; P = .01).⁷

Of those RCTs eventually published, the full-length journal article was much more likely than the abstract to detail the methodology of the RCT, the sources of funding, and the trialists' affiliations (Table 2). In 35 of the 148 abstract/manuscript pairs (24%), the sample size differed between the abstract presented at ACC and the eventual full-length publication in 24 cases (16%), the full manuscript reported a larger sample size (median increase 21 patients; interquartile range, 6-209 patients), and in 11 cases (7%), the full-length article reported a smaller sample size (median decrease 6 patients; interquartile range, 2-97 patients). In 10 of the abstract/manuscript pairs, the sample size discrepancy exceeded 10% (4 of the 79 [5%] late-breaking abstract/manuscript pairs and 6 of the 69 [9%] RCTs presented at other sessions; P = .52). In 6 of these cases, the sample size discrepancy arose because the meeting abstract reported preliminary results, in 1 case the study participant eligibility criteria described in the abstract and the full-length article were different, in 1 case the full-length article reported a subsample from the abstract data, and in 2 cases the reasons for the sample size discrepancies were not apparent.

Many of the treatment effect estimates changed from the ACC abstract to the full-length published article. Sixty (41%) of the 148 RCTs that were eventually published exhibited a change in the point estimate between the meeting abstract and the full-length publication (in 20 of these cases the effect estimate was statistically significant in only 1 of the pair). The discrepancy rate was almost identical between RCTs presented in the late-breaking sessions and RCTs presented in other ACC sessions (Table 3). When we computed the magnitude of the changes using standardized effect sizes, the mean change in effect was 0.44 SDs when all pairs were examined and 1.01 SDs when only the changed pairs were evaluated (Table 3). For the abstract/manuscript pairs that did differ, the non−late-breaking RCTs exhibited greater changes in effect estimates between the meeting abstract and the eventual full-length publication than the late-breaking RCTs (P = .007; Table 3).

In summary, RCTs presented at the late-breaking clinical trials session at the ACC are larger, are more likely to be preceded by published design papers, and are less likely to report results favorable to the intervention than RCTs presented at other sessions of the ACC. They are also more likely to be published as full-length journal articles (especially in journals with high impact factors) and are published sooner after presentation (even after adjustment for differences in other factors affecting publication rate and speed, such as sample size, RCT design, and study conclusions). Our study has established that presentation at the late-breaking trials session of a scientific meeting should be added to the list of factors that are associated with enhanced publication rates.^1,9 However, discrepancies between meeting abstracts and subsequent full-length journal articles occurred commonly, even for late-breaking trials—the effect estimate for the primary outcome differed between meeting abstract and subsequent full-length publication in nearly half of all cases and in almost one seventh of the abstract/manuscript pairs, the results were statistically significant in only 1 member of the pair.

The publication rates we observed (92% for late-breaking RCTs and 69% for other RCTs) are higher than the figures cited in the Cochrane review of noncardiology studies (45% for all meeting abstracts and 58% for RCTs).¹ The extent to which this higher publication rate for abstracts presented at the ACC is due to the strength of the ACC meeting in attracting high-quality clinical research or a bias of journals in favor of cardiology over noncardiology topics is unknown and cannot be determined from our study. Similarly, it is impossible to say whether the greater publication rate of late-breaking trials is due to the fact that the ACC meeting organizers were successful in identifying RCTs that would have broad appeal or whether the publicity attendant with the late-breaking trials session at ACC stirred sufficient editorial interest to propel these RCTs into high-ranking journals.

The process by which RCTs are selected for the late-breaking session may account for some of the differences we found between these RCTs and RCTs presented at other ACC sessions. For example, as investigators must apply to the program committee of the annual ACC meeting to present at the late-breaking trials session and report the purpose, design, and methods of their trial on the application form 3 months before the meeting (as described at http://www.acc.org/2006ann_meeting/abstract/lbct.htm), it is not surprising that late-breaking trials are larger and more likely to have a published design paper than other RCTs. Further, as the results of late-breaking trials are not known at the time they are selected for presentation, it is also not surprising that they were less likely to report favorable results for their tested interventions than RCTs presented in other ACC sessions (where the conventional biases in favor of studies reporting positive results are more likely to be operant). However, the selection process for late-breakers should have no influence on subsequent discrepancies between the data presented at the meeting and the data reported in subsequent full-length publications. Indeed, as late-breaking trials are asked to present final data in so far as is possible at the time of presentation (and increasingly are published online at the same time as they are presented), we expected late-breaking trials to exhibit more concordance between meeting abstracts/presentations and full-length publications than RCTs presented at other sessions; however, we found frequent inconsistencies between the data presented in meeting abstracts and the corresponding full journal publications and, importantly, similar discrepancy rates for late-breakers vs other RCTs.

While our findings are consistent with previous studies of abstracts from surgical and pediatric meetings,^4- 6 and a study documenting that between 18% and 68% of published articles contained inconsistencies between the abstract and the text within the same manuscript,¹⁰ we extend this literature by exploring the consistency between abstracts and subsequent publications for RCTs, including those selected for presentation at late-breaking trial sessions. Taken together with the lessons from the carotid endarterectomy literature (in which prepublication release of the NASCET¹¹ and ACAS¹² trial results led to rapid and substantial increases in carotid endarterectomy rates but in cases and settings that would have been ineligible for the trials¹³), our study supports continued caution in the incorporation of results from meeting abstracts into clinical practice, meta-analyses, or practice guidelines. In the words of Fontanarosa and Flanagin, the decision to implement a particularly promising intervention should wait until the study has been “evaluated by rigorous peer review for scientific validity and clinical importance.”¹⁴ Indeed, we question whether the current enthusiasm to simultaneously publish late-breaking trials online at the time of presentation may attenuate the opportunity for incorporating feedback from expert colleagues both during and subsequent to the presentation and preclude the sober second thought so useful in placing trial results into perspective. Regardless, given recent data suggesting that one sixth of interventions reported to be beneficial in widely cited studies were found to be ineffective when larger or more rigorous studies were done, perhaps caution is warranted even after the initial full-length publication in support of a new intervention.¹⁵

Although we conducted hand searches to identify all RCTs presented at a major international cardiology meeting, systematically searched for subsequent publications, and double-extracted all data, our study is not without limitations. For one, we did not contact abstract authors for studies for which we were unable to find full-length journal publications to determine if their abstract was ever published; however, we used extensive literature searches (including gray literature), which previous studies in this field have not done (of the 73 previous studies examining publication rates of abstracts, most searched only MEDLINE).¹ Second, we acknowledge that the publication rates we report (although already higher than the rates cited in previous studies) will likely underestimate the final publication rates for these abstracts since we only followed the abstracts for 3.5 (the 2002 ACC abstracts) to 6.5 (the 1999 ACC abstracts) years. However, previous studies have shown that the median time to publication for RCTs is approximately 18 months and the publication rate slows substantially after 3 years.¹ Third, we relied on published reports for both the abstracts and the full-length published articles and acknowledge that, particularly for judging study quality, this is imperfect. Finally, we recognize that preliminary RCT presentations at scientific meetings may focus on preliminary data for end points other than the primary outcome and for this reason, we restricted our analysis to the meeting abstract and full-length journal publication for each included RCT that discussed the primary outcome for that RCT.

In conclusion, we have answered the question of what happens to RCT abstracts after presentation at a major international scientific meeting and, in particular, those RCTs presented at the late-breaking trials sessions. Although most RCTs presented at the ACC meeting are subsequently published, especially those presented at the late-breaking trials sessions, discrepancies between meeting abstracts and subsequent full-length journal articles are not uncommon. Our findings should sound a cautionary note for those advocating widespread adoption of an intervention on the basis of a scientific meeting presentation without waiting for the full-length peer-reviewed publication. As Shakespeare may have said in the 21st century, there are many slips betwixt podium and page.