Relation Between Prepublication Release of Clinical Trial Results and the Practice of Carotid Endarterectomy FREE

Health policymakers, medical journal editors, scientists, and the lay press have debated whether it is in the public's best interest to release the results of clinical trials before the methods and results have been peer-reviewed and published.^1- 5 Medical journal editors usually prohibit widespread dissemination of clinical trial results until a manuscript is published, emphasizing the importance of the peer-review process.^1,6 This ensures that clinicians, patients, and researchers have nearly simultaneous access to pertinent information, such as the characteristics of the trial participants, the nature of the intervention, and the setting in which it was applied.⁵

Members of the lay press, however, have argued that the public has the right to know study results as soon as they are available, especially if the trials have been funded by a governmental agency such as the National Institutes of Health (NIH).⁴ Since there can be a delay of up to 6 months between acceptance of a manuscript by a journal and subsequent publication, it has been suggested that dissemination of clinical trial results prior to publication may facilitate adoption of new information in a more expedient manner.

In response to these concerns, the prepublication embargo has been waived in special circumstances, such as when trial results could have a significant impact on patient care.^1,6- 7 When such trials are sponsored by the NIH and the institute director determines that there is sufficient medical urgency and medical importance, the National Library of Medicine releases a clinical alert to notify the public at risk.^8- 9 It is unclear whether patterns of care change quickly enough to merit this rush to disseminate information and whether results described in clinical alerts are applied to patients for whom benefits have been demonstrated.

We therefore analyzed the relationship between 2 clinical alerts and clinical practice at the population level. Each clinical alert described the results of a clinical trial that favored the use of carotid endarterectomy (CEA). We investigated whether dissemination of the alerts was associated with (1) a prompt change in the use of CEA, (2) a change in use of CEA in patients who met the age criteria of the studies in comparison with those who did not, and (3) a change in the use of CEA at hospitals with high vs low-operative mortality. We also examined whether subsequent journal publication of the full article was associated with further changes in the pattern of CEA use.

We performed a longitudinal analysis using hospital discharge data from patients who received a CEA in 7 states from 1989 (2 years before the NIH released a clinical alert on the North American Symptomatic Carotid Endarterectomy Trial [NASCET]) to 1996 (2 years after the clinical alert on the Asymptomatic Carotid Atherosclerosis Study [ACAS]).^10- 13 The important events in the dissemination of the results of each trial are listed in Table 1. The purpose of our study was not to assess the appropriateness of CEA at the population level; previous studies have demonstrated that CEA is frequently used in clinical settings that are considered inappropriate by national experts.^14- 16 Rather, we focused on how the patterns of CEA use changed after dissemination of trial results; we considered both the overall utilization rate and the use of CEA in specific patient groups or hospitals that were not included in the trials.

The NASCET study was a randomized trial of patients younger than 80 years with symptomatic carotid artery stenosis (ie, mini-stroke or nondisabling stroke 90 days prior to entry). The study was stopped early in patients with 70% to 99% stenosis, because evidence of efficacy was attained for patients in this subgroup who were randomized to CEA (absolute risk reduction in ipsilateral stroke over an 18-month follow-up, 17%).¹⁰ A clinical alert was issued informing clinicians of these results in February 1991, and the article was published in the New England Journal of Medicine 6 months later.^10,12 The clinical alert recommended that CEAs should only be performed at centers with "documented expertise," whereas the article specifically stated that "if perioperative risk of major stroke and death exceeds the 2.1% reported here, the benefits of CEA will diminish."^10,12

The ACAS enrolled patients with asymptomatic carotid artery stenosis of greater than 60% who were aged 40 to 79 years and randomized them to receive either CEA or medical care.¹¹ This study also was stopped early, because of a significant decrease in stroke in the CEA group (absolute risk reduction, 5.8%) after a median follow-up of 2.7 years.¹¹ In September 1994, the NIH issued a clinical alert advising clinicians of the results.¹³ The full article was subsequently published in JAMA in May 1995.¹¹ In both the clinical alert and the article, the investigators cautioned that results may be applicable only at facilities with perioperative morbidity and mortality rates below 3%.^11,13

We also examined whether use of CEA changed prior to publication of a large randomized trial that was not preceded by a clinical alert. The Veterans Affairs Cooperative Study Group¹⁷ published the results of a trial involving patients with asymptomatic carotid artery stenosis in January 1993. After a mean 4-year follow-up, the authors reported that patients in the surgical group who received a CEA were found to have a decreased risk (absolute risk reduction, 12.6%) of neurologic events (stroke and transient ischemic attack [TIA] combined).¹⁷

We used data from 7 states (New York, California, Pennsylvania, Florida, Colorado, Illinois, and Wisconsin) that participate in the State Inpatient Databases (SID) of the Healthcare Cost and Utilization Project (HCUP).¹⁸ States were selected for representation of state size and geography. The CEA utilization rate was calculated for all 7 states in aggregate. The total population included in our analysis was approximately 97.2 million people in 1996,¹⁹ representing approximately 37% of the US population. Similarly the total number of hospital discharges in these states represented 36% of all US discharges.¹⁸ The SID contains uniform, comprehensive patient-level clinical and resource-use data on inpatient stays in acute care nonfederal hospitals. We extracted records for all patients who had received a CEA, identified by International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM)²⁰ procedure code 38.12.

We also extracted data on all patients who received either abdominal aortic aneurysm (AAA) surgery (ICD-9-CM codes 38.34, 38.44, or 38.64) or coronary artery bypass graft (CABG) surgery (ICD-9-CM codes 36.10-36.16, 36.20, and 36.30) to assess secular trends in the use of vascular procedures.

We used direct adjustment to calculate the age- and sex-adjusted CEA rate among persons aged 40 years or older, using the US 1990 population as the standard.¹⁹ We used linear regression to determine whether the age- and sex-adjusted CEA rate increased in months subsequent to the clinical alerts and after each journal publication.²¹ The dependent variable was age- and sex-adjusted CEA rate, while admission month was the independent variable. We were interested in contrasting the rate of change in the CEA rate across several periods (ie, the period prior to clinical alert dissemination vs the period between the clinical alerts and article publications). We used linear spline functions to allow estimation of the relationship between CEA rate and month in a piecemeal linear function, with straight lines representing the discrete periods.²¹ Additionally, age- and sex-adjusted AAA and CABG procedure rates were calculated and the analysis was repeated using each of these rates as dependent variables.

Because both the NASCET and the ACAS trials excluded patients aged 80 years or older, we repeated the analysis after stratifying by age group (50-79 vs 80 years). We limited this analysis to patients older than 49 years, because this group comprised about 98.5% of CEA cases in our sample. We used Zellner seemingly unrelated regression to determine whether the clinical trial results affected use of CEA differently in older vs younger patients.^22- 23 This technique allows comparison of models that have the same independent variables, allowing for potential correlation of the error terms.²² For each subgroup, we performed 2 related regressions. Specifically, we used both the number of CEAs in older patients and the number in younger patients as dependent variables. We then used a linear test of hypothesis to determine whether there was a statistically significant difference between the parameter estimates derived in the separate equations.

A major caveat of the NASCET and ACAS trials was that CEAs should be performed in hospitals with low risk of perioperative morbidity and death. Hence, we stratified the hospitals to examine whether those with a high volume and low in-hospital CEA mortality rate experienced a proportionately greater increase in CEAs after the clinical alerts. The time surrounding the dissemination of each trial was divided into 3 discrete periods. The 2-year interval prior to each clinical alert was defined as the before clinical alert period and the after clinical alert was the period between each clinical alert and the corresponding article publication; after publication referred to the interval after publication.

Using the same criteria as the NASCET investigators, who excluded hospitals that had performed fewer than 50 CEAs in the 2 years before the study, we defined hospitals with fewer than 50 CEAs in the before clinical alert periods as low-volume centers. The remaining high-volume centers were further stratified into low-mortality and high-mortality sites, based on their in-hospital CEA mortality rates in the before clinical alert periods. We defined a low-mortality hospital as one in which the in-hospital mortality rate was less than 2%, as was done in a prior study.¹⁴ The proportion of CEAs that were performed at each type of hospital in each period was calculated, and we used a χ² test to compare periods.

From 1989 through 1996, a total of 272,849 CEAs were performed in the 7 states' nonfederal hospitals, with the annual number increasing from 22,300 to 51,495. Figure 1A shows the age- and sex-adjusted CEA rates during each period and their associated regression lines. The CEA rate did not change significantly during the 2 years prior to the NASCET clinical alert, averaging approximately 5.35 CEAs per 100,000 persons aged 40 years or older.

After the release of the NASCET clinical alert in February 1991, the CEA rate increased approximately 3.4% each month (95% confidence interval [CI], 1.6%-5.3%) during the next 6 months (Table 2). During the after clinical alert period, the CEA rate had increased 18% from the baseline (from 5.35 to 6.34 per 100,000 persons aged 40 years or older; P<.001). Although the use of CEA continued to increase after NASCET was published, there was a significant decline in the rate of increase of CEA(0.5% per month; 95% CI, 0.2%-0.8%; P = .007 for test of difference in trend between periods).

Following the ACAS clinical alert in 1994, the CEA rate increased an average of approximately 7% each month (95% CI, 6.0%-8.5%) for 7 months, with a total increase of 42% (from 7.5 to 10.7 per 100,000 persons aged 40 years or older; P<.001). After publication of the ACAS article, there was a significant decrease in the CEA rate (0.44% per month; 95% CI, −0.86% to −0.0002%). The relationship between the CEA rate and each of the various periods was consistent in each of the 7 states (data not shown).

We repeated the analysis using the 6-month period immediately prior to the release of the Veterans Affairs Cooperative Study in January 1993.¹¹ Unlike NASCET and ACAS, this Veterans Affairs study was not preceded by a clinical alert. We found no significant change in rates of CEA use during the 6-month period prior to publication of that article (P = .07).

Although the CEA rate among persons aged 50 to 79 years was initially similar to the rate among persons aged 80 years or older, there was a statistically significant higher rate in the group aged 80 years or older by the end of the study period with a mean (SD) of 15.34 (1.03) for the younger group and 21.57 (2.03) for the older group (t test, P<.001) (Figure 1B). The in-hospital mortality rate for the older group (1.93%; 95% CI, 1.78%-2.07%) was significantly higher than the mortality rate in the younger group (1.14%; 95% CI, 1.09%-1.18%) throughout the study period.

Because the use of CEA increased at different rates in each age group, we estimated the incremental change in CEA use for each period, stratified by age (Table 3). After the NASCET clinical alert, the CEA rate among persons aged 50 through 79 years increased by an additional 0.25 procedures (95% CI, 0.09-0.42) per 100,000 persons each month aged compared with the baseline rate before the clinical alert. In comparison, the rate among persons aged 80 years or older increased by about 0.38 CEAs per 100,000 persons each month (95% CI, 0.13-0.64). Although there was a trend toward a greater increase in the CEA rate in the older age group, the difference was not statistically significant (P = .08).

In both age groups, the rapid increase in the use of CEA after the clinical alert was no longer apparent after publication (Figure 1B). The average monthly change in CEA use was 0.23 CEAs per 100,000 less in the younger group after publication than it was before publication. The group aged 80 years or older experienced a similar deceleration in CEA rate; the average monthly change in this group was approximately 0.33 CEAs per 100,000 lower after publication (Table 3). There was no significant difference between the age groups regarding the magnitude of this deceleration (P value for difference = .14).

After the ACAS clinical alert, there was a significantly greater increase in CEA use in the older age group than there was in the younger one (increase of approximately 0.95 vs 0.73 cases per 100,000 persons per month, respectively; P value for difference <.001). Conversely, after the ACAS article was published, the rate of change slowed dramatically for both age groups, but the deceleration was significantly greater in the older group (Figure 1B; P<.009).

Although the overall use of CEA increased dramatically after both clinical alerts, patients were not preferentially referred to low-mortality hospital centers; rather, the proportion of patients that were referred to each type of hospital stayed relatively constant (Table 4). After the NASCET clinical alert was released, the proportion of CEAs that were performed at high-volume, low-mortality hospitals did not change significantly (50% before the clinical alert vs 49.4% after). After NASCET was published, the proportion of CEAs performed at low-mortality hospitals decreased slightly (to 48.4%, P<.001 for comparison with the proportion prior to the clinical alert).

Similarly, the proportion of CEAs that were performed at high-volume, low-mortality hospitals decreased slightly after the ACAS clinical alert (from 64.5% to 63.8%; P = .04), and after publication of the ACAS article (to 62.4%). In contrast, there was a significantly greater increase in the use of CEA in low-volume centers after the dissemination of each trial (P<.001 for both comparisons; Table 4).

Although there was an overall downward trend in the number of AAA surgeries performed during the study period, our regression analysis confirmed that after each CEA clinical alert, there was no significant change in this downward trend (Figure 2). The utilization of CABG increased steadily during the entire study period. After the ACAS clinical alert, there was a significant increase in the CABG rate compared with the rate in previous periods. However, the CABG rate increased only about 1.4% per month (95% CI, 0.6%-2.2%), compared with the CEA rate, which increased 7.3% per month during this interval.

Journal editors have been criticized for not routinely allowing prepublication release of information that may "have an immediate benefit for the public."^4,24 However, some have questioned whether new information can change patient care in a prompt and meaningful way.²⁵ Only a handful of studies have demonstrated that randomized controlled trials were associated with a measurable change in practice patterns.^26- 29 In fact, some evidence has suggested that the time interval between the publication of clinical trial results and the actual adoption of new information by practitioners can be as long as 12 years.³⁰

This population-based study suggests that the prepublication release of these trial results was associated with a prompt and substantial change in clinical practice. The data are consistent with other studies demonstrating that NASCET and ACAS were associated with increased utilization of CEA following publication.^14,31- 32 Unlike these prior studies, our data specifically address the mode of dissemination. Each clinical alert was disseminated about 6 months prior to publication and was associated with a rapid increase in the use of CEA.

However, "substantial changes in practice" are not uniformly beneficial. It is important to note that these particular studies occurred under ideal settings and with excellent surgeons. For the ACAS study, for instance, about one third of the surgeons who applied to participate either did not complete the certification process or were not approved.³³ The perioperative mortality rate of these surgeons was 2.25%, a rate that was 22 times higher than the actual rate in the study.³³ This rate was comparable with another population-based study of CEA outcomes in hospitals with average volume (1.9%).³⁴ In that study, the perioperative mortality rate was even higher in patients 84 years or older (3.6%).³⁴

The ACAS investigators were clearly concerned about extrapolating their results to settings in which the operative risk was much higher. The summary statement of the clinical alert states that "it is important to note that the success of the operation is dependent on medical centers and surgeons who have a documented perioperative morbidity and mortality of less than 3 percent. . . . "¹³ The same caveat was repeated in the abstract and text of the ACAS article.¹¹

Although the NASCET clinical alert contained no such warning, the article included the following statement: "We caution readers not to apply our conclusions too broadly. First the study surgeons were selected only after audits . . . confirmed a high level of expertise. If comparable expertise and quality control are not achieved in widespread implementation of these results, . . . the benefit of CEA will diminish."¹⁰

Clinical investigators seldom are so frank about warning clinicians not to extrapolate the results of their study to inappropriate settings. Our data demonstrate that, at least with regard to hospital expertise and patient age, these warnings apparently went unheeded. There was a substantial increase in CEA use in the type of hospitals in which the authors were trying to discourage. Similarly, although it is well-known that elderly patients have increased surgical risk, there was a large increase in CEA rates in this age group.

As a result, the expected health benefits of prepublication release may have been limited because it appears that 2 of the trials' important exclusion criteria did not guide clinical application of the study results. Although both studies were limited to patients younger than 80 years, the increase in CEA use was actually greater in older adults after the ACAS clinical alert. Conversely, following publication of the ACAS trial, the relative decrease in CEA rate was significantly greater in the older age group. The ACAS article included a subgroup analysis that suggested that the impact of CEA on stroke reduction was statistically significant in patients younger than 68 years (relative risk [RR], 0.60; 95% CI, 0.11-0.82), but it was not significant in patients aged 68 through 79 years (RR, 0.43; 95% CI, −0.07 to 0.70).¹¹ This analysis had not been included in the preceding clinical alert.¹³ Thus, after the evidence in the full article was available for review, clinicians may have been increasingly reluctant to recommend CEA for older patients.

In our study population, the risk of in-hospital mortality for patients 80 years or older was approximately 50% higher than the risk for CEA patients younger than 80 years. Previous authors have pointed out that elderly patients not only have a lower lifetime risk of stroke than their younger counterparts, but they also have a lower threshold of acceptable perioperative mortality and morbidity with CEA, above which the risks of surgery outweigh the benefits.^35- 36 Thus, it is possible that increased use of CEA may cause more harm than benefit in some elderly patients.

Our data also indicate that clinicians did not selectively refer their patients to high-volume, low-mortality hospital centers, even though the clinical alerts included the caveat that CEAs should be performed in centers having "documented surgical expertise."^10,12 Unlike patient age, which is relatively easy to incorporate into clinical decision making, there are few data available to clinicians or to the public regarding hospitals' experience or complication rates.³⁷ As a result, many patients may not have derived the full benefit of CEA, given that they were referred to centers with higher operative mortality rates. Future studies should explore whether widely accessible data on complication rates will enable more physicians and patients to choose hospital centers with lower surgical risk.

The trend in CEA use appeared to plateau after each article was published. Because the largest increase in CEA rate occurred in the first few months after each alert, it is possible that this plateau in CEA use may have been unrelated to article publication. The CEA rate may have reached a new baseline level after the clinical alerts regardless of the articles' contents or clinicians' reaction to them. Alternatively, some clinicians may have read the articles or the subsequent critiques, and enthusiasm for CEA may have decreased after they had the opportunity to scrutinize closely the complete articles. Because experts questioned the generalizability of the results, the published articles met with considerable skepticism.^38- 41

Our study had several limitations. We had no data on the clinical indication for surgery, so we were unable to investigate whether the NASCET and ACAS trials influenced CEA utilization primarily in symptomatic and asymptomatic patients, respectively. Additionally, this study did not seek to address whether CEA was appropriate for each patient; our focus was on 2 important inclusion criteria for the studies in question.

It is important to note that the observed associations between the dissemination of these clinical trials and a change in clinical practice may not be causal. However, several important criteria for the determination of causality have been met. These include plausibility, a strong temporal relationship (demonstrated by a prompt increase in CEA rate after the clinical alerts), a strong association between the factors in question (demonstrated by a substantial increase in rate), replication in different settings and populations (similar change was noted after both clinical alerts in each of the 7 states), and specificity, with regard to type of procedure (findings were specific to CEA, compared with CABG or AAA).⁴² Additionally, we did not find an increase in CEA use prior to the Veterans Affairs study, which was not preceded by a clinical alert. However, the lack of response to the Veterans Affairs study may also relate to the fact that the results of this study were less impressive than either the NASCET or ACAS study; the Veterans Affairs investigators were only able to demonstrate a significant impact of CEA when TIA was combined with stroke as an outcome measure.¹⁷

There were several characteristics unique to the ACAS and NASCET trials that likely rendered them particularly effective in influencing physician practice. Each trial was presented at several national meetings, and the news of the results was reported in numerous prominent newspapers. The national meeting presentations all occurred after the clinical alerts were released; however, they served to build on the prepublication dissemination process that was initiated by the alerts. Moreover, the clinical alerts were transmitted directly to a wide pool of clinicians via the National Library of Medicine's online MEDLARS network, which has more than 40,000 users. Copies of the alert also were mailed to all 130 medical school libraries in the United States, as well as to 3600 members of the National Library of Medicine's national library network. Additionally, the results of these trials may have been more eagerly anticipated than most due to the controversy surrounding the use of CEA. This may have affected adoption, as it has been suggested previously that changes in use of technology are influenced by the receptiveness of physicians to new information.^37,43 Additionally, findings in the 7 study states may not be generalizable to all states. Finally, although the CEA rate was initially similar for the elderly and younger age groups, by the end of the ACAS clinical alert period, the rate was substantially higher in the older group. Hence, the greater relative decrease in CEA rate noted in the older group may represent regression to the mean.

We conclude that dissemination of clinical trial results prior to publication was associated with a prompt and substantial change in the use of CEA. However, policymakers should consider that the prepublication release of clinical trial results may have unintended consequences. The new medical evidence was not necessarily applied in the types of patients studied by the authors or in hospitals with the recommended expertise. Because publication of the full article may help promote optimal use of the new information, future work should explore the impact of other mechanisms of prepublication dissemination, such as Internet publication of full-length, peer-reviewed articles.⁴⁴