Effect of Awareness of a Randomized Controlled Trial on Use of Experimental Therapy FREE

The randomized controlled trial (RCT) is considered the gold standard to establish the effects of a therapeutic intervention^1- 3 whenever the level of uncertainty about its efficacy makes a trial both ethical and practical.^4- 8 There already is ample evidence that the results of completed RCTs influence the subsequent use of the experimental therapy,^9- 14 with positive and negative studies followed by positive and negative shifts in the use of the experimental therapy.^11- 12 For example, 2 recent inquiries into subsequent use of lipid-lowering agents following publication of major positive RCTs documented significant positive shifts in prescription of lipid-lowering drugs^13- 14; one study also found a significant positive shift in the market share of the specific drug used in the trial.¹⁴

In many cases, the experimental intervention is already available and in use prior to an RCT's initiation, but our inquiries and literature search failed to unearth any studies that documented the effect of awareness that a trial was under way on concurrent use of the experimental therapy. We therefore decided to extend these posttrial inquiries backward in time to determine if use of an experimental therapy increases during an RCT evaluating the therapy among patients in both trial and nontrial centers.

Our inquiry was possible because the Canadian Apheresis Group, comprising 39 apheresis units in all 19 major medical centers in Canada, operates a national registry for therapeutic apheresis including the clinical indications for each apheresis. The registry has been in operation since 1980 and is presumed to capture 90% of apheresis activity. In Canada, the "gatekeeper" function for apheresis is a joint decision made by the Canadian Apheresis Group and a specialty physician, a practice that predates all of the RCTs we studied and that is not affected by health maintenance organizations. In fact, not all Canadian Apheresis Group physicians and their centers were involved in the RCTs studied. Some Canadian Apheresis Group physicians elected not to obtain human ethics approval; therefore, their centers were excluded from trial participation. Among the 3 apheresis RCTs that we studied, the ongoing myeloma cast nephropathy study provided an opportunity to assess behavior within known trial and nontrial centers with respect to their activity prior to onset of the trial and during the trial. In the thrombotic thrombocytopenic purpura (TTP) and myeloma cast nephropathy studies, the principal investigators were Canadian Apheresis Group physicians, but even in these centers, although almost all patients were entered into RCTs, there were exceptions.

The Canadian Apheresis Group has sponsored major national peer-reviewed RCTs on the use of apheresis in multiple sclerosis, TTP, and myeloma cast nephropathy.^15- 16 For 1 to 2 years prior to each RCT, Canadian Apheresis Group physicians discussed clinical anecdotes and the inconclusive results of prior flawed trials and, when uncertain about the efficacy of apheresis, formed committees (to which were added hematologists, neurologists, nephrologists, and oncologists as necessary) to develop RCT protocols, present them to the entire Canadian Apheresis Group membership for approval, and obtain national peer-reviewed funding. Although Canadian Apheresis Group physicians and their relevant referring physicians participated in designing these trials, to avoid study bias, there were no formal attempts to inform family physicians about these trials or to alter their referral patterns during these RCTs, no media interviews or advertisements about them, and no interim analyses reported to Canadian Apheresis Group members.

Data from the Canadian Apheresis Group registry were extracted to document the use of apheresis by clinical indication before, during, and after the completed RCTs in multiple sclerosis (1986-1988) and TTP (1982-1988) and before and during the ongoing RCT in myeloma cast nephropathy (1998-2000). The data allowed a comparison of apheresis for the trial conditions and all other disease processes. The end of a trial was defined as the date on which its results were announced to all Canadian Apheresis Group physicians. The ongoing nature of the myeloma cast nephropathy study allowed an annual center-by-center assessment of use of the experimental therapy in both trial and nontrial centers prior to and during the RCT. In contrast, the completed TTP and multiple sclerosis trials did not indicate the individual sources of the data.

Changes in the numbers of patients undergoing apheresis prior to and during the multiple sclerosis and myeloma cast nephropathy RCTs were assessed by a time-series analysis using interactive, autoregressive, integrated moving average models to determine the impact of initiation of each RCT on frequency of apheresis use for multiple sclerosis or myeloma cast nephropathy.¹⁷ The Canadian Apheresis Group was formed in 1980; therefore, we had data for only 1 year prior to the initiation of the TTP study and so could not carry out a time-series analysis of this information.

Twenty-four Canadian Apheresis Group physicians at the 19 trial and nontrial centers were sent a questionnaire to determine whether they had perceived any increase in apheresis during these trials and, if so, their explanation for it. A follow-up questionnaire gave them the results of our analysis and asked them whether they thought that a "jumping-the-gun" phenomenon explained the increase in nontrial apheresis activity.

During all 3 RCTs, there were large increases in use of apheresis, as shown in Figure 1, for multiple sclerosis (P<.001), TTP, and myeloma cast nephropathy (P = .004).¹⁷ As shown in Table 1, a substantial portion of this increase occurred outside of these trials. Although the use of apheresis for multiple sclerosis more than tripled during its trial (from 13 patients per year before the trial to 49 patients per year during the trial; P<.001), the majority of multiple sclerosis patients undergoing apheresis during the trial (30 of the 49 patients per year; 61% of the increase) were treated outside the trial. Similarly, the number of TTP patients undergoing apheresis increased from 30 patients per year before the trial to 56 patients per year during the trial, but most of the latter (49 of the 56 patients per year; 72% of the increase) were treated outside the trial. Finally, although the number of myeloma cast nephropathy patients undergoing apheresis increased from 44 patients per year before the trial to 72 patients per year during the trial (P = .004), most of the latter (60 of the 72 patients per year; 57% of the increase) were treated outside the trial. Moreover, during 3 years of the myeloma cast nephropathy trial, the increased use of apheresis for this condition was documented in both trial and nontrial centers. The number of patients undergoing apheresis in trial centers increased from 62 to 127, of which 35 patients (54% of the increase) were enrolled in the RCT. Even in nontrial centers, the number of patients increased during the RCT from 71 patients to 88 patients, representing a 19% increase. The increases in all trials were far greater than could be explained by a general secular increase in apheresis for all conditions, as noted in Figure 1.

In the multiple sclerosis trial, there was no evidence that inclusions for trial entry systematically excluded patients or placed them into the "outside the trial" category. However, in the TTP trial, there were approximately 3 patients per year over the 7 years of the trial who were categorized as outside the trial because of insufficient renal function to be randomized into the plasma infusion arm of the trial. In the myeloma cast nephropathy trial, 2 patients were categorized as outside the trial because their age exceeded 80 years, which is one of the exclusion criteria for the study. Thus, a small proportion of the "outside the trial" patients had been considered for the trial but rejected.

The study awareness phenomenon occurred for a disorder in which the respective apheresis trial ultimately showed this treatment to be beneficial (TTP) and for a disorder for which it was judged to be useless (multiple sclerosis).^15- 16 In the TTP trial, patients receiving plasma exchange had a 50% reduction in mortality compared with those receiving plasma infusion, and in the multiple sclerosis trial, there was no benefit noted by blinded observers in disability score for those who received apheresis vs those who did not. After the positive TTP trial, results were announced and published and the use of apheresis for this condition continued to increase as evidenced by Figure 1, C, and as noted in previous updates.^12,16 Apheresis for multiple sclerosis rapidly returned to low pretrial levels when it was judged useless for this condition, as noted in Figure 1, B, and also in our previous updates of apheresis activity.^12,15

The Canadian Apheresis Group physician survey had 22 respondents from the 19 medical centers (92% response rate), and 12 respondents had noticed an increase in apheresis during the RCTs, which accords with observed Canadian Apheresis Group registry data. The most popular explanation for the increase in apheresis activity during RCTs, noted by 7 of the 12 respondents, was increased referral for the procedure; however, only 2 respondents attributed it to an increased demand for apheresis by patients. On the follow-up survey, 18 apheresis physicians who had responded to the initial survey and had participated in at least 1 of the RCTs were asked if they thought this jumping-the-gun phenomenon reported on the initial survey explained the increase in apheresis activity noted at their center during the 3 RCTs. Three strongly agreed, 11 agreed, 2 had no opinion, and 2 disagreed. Thus, 78% of responding physicians agreed that the jumping-the-gun phenomenon may explain the increase in apheresis activity noted at their center during 3 RCTs.

During 3 multicenter RCTs, an increase in the number of patients receiving experimental therapy (apheresis) was documented. This increase occurred in both participating and nonparticipating centers, and the majority of the increase was among nontrial patients. Moreover, this increased use of apheresis occurred both during trials that eventually showed its efficacy and in 1 that concluded that it was useless. Jumping the gun by applying an unproven therapy during RCTs of its efficacy may be a real phenomenon.

We are not certain of why jumping the gun occurs, and our study permits only speculation about its explanation. It is tempting to attribute jumping the gun to decreases in physician uncertainty about the efficacy of an experimental intervention when it becomes the object of a randomized trial.^18- 19 During the 1 to 2 years of preparation prior to each RCT, Canadian Apheresis Group and subspecialty physicians were contacted to help in the identification of literature relevant to the research question posed. This process increased physician awareness of the possible efficacy of the experimental therapy (apheresis) in the disease under study. However, if this process alone was responsible for increased use, we should have observed it during the planning phases of these RCTs, but we did not.

Is it possible that many clinicians do not understand the uncertainty principle? The inclusion of new treatments in randomized trials may legitimize them in their minds as treatments worth trying in patients who have more severe forms of the target disease, who perhaps have failed to respond to other treatments, and who are viewed as having no alternative options. Existence of a national, peer-reviewed, funded trial signals that there is an enthusiastic group of practitioners who can provide the new treatment and whose patients may "deserve" to try it. The enthusiasm for the new treatment that is being studied may depend in part on the perception of the likely trade-off between benefit and harm. In the case of apheresis, it is often viewed as having relatively little risk; thus, clinicians may refer patients on the grounds that it may not help but that "it won't do any harm."

The observed increase in the number of patients receiving apheresis during each RCT is the result of both greater number of referrals by the community of specialized interest and greater acceptance for treatment by Canadian Apheresis Group physicians. The majority of surveyed Canadian Apheresis Group physicians noticed this increase during the 3 RCTs, and their most popular explanation was an increase in patient referral during the trials. If this is true, it suggests that both referring and Canadian Apheresis Group physicians (despite denial from the latter) may become more certain about the possible benefits of an experimental therapy based on their interpretations of its effects during its trial. As in the multiple sclerosis trial with results showing no effect, these interpretations can be incorrect and are not a substitute for randomized evidence. There was little evidence that patients receiving apheresis outside the trial failed to meet eligibility criteria for entrance to the trial, except for 3 patients per year in the TTP trial and 2 patients in total in the myeloma cast nephropathy trial.

Other possible explanations were not supported by our data. If, for example, other high-quality evidence of the efficacy of apheresis for the trial condition became available during these trials, an increased use of the experimental therapy during them would no longer represent jumping the gun but simply practicing up-to-date medicine. However, no external published evidence of efficacy appeared during any of the trials we examined. Alternatively, it is possible that increasing patient preference for an experimental therapy could increase its use. In other words, patients who are apprised of the role of apheresis as part of the potential RCT entry process could elect not to participate and instead demand apheresis treatment. This in part might explain the greater increase of apheresis during the myeloma cast nephropathy study in patients outside the RCT in trial vs nontrial centers. However, there was an increase in apheresis for myeloma cast nephropathy in nontrial centers at a time when overall apheresis activity for all other conditions was stable (Figure 1, D). This nontrial center increase in apheresis, not mirrored by an overall increase in apheresis activity, is in keeping with a jumping-the-gun phenomenon. This observation is strengthened by the fact that only 2 of 22 Canadian Apheresis Group physicians indicated in the questionnaire that many of their eligible patients insisted they receive apheresis. There was no evidence that patients eligible for treatment were excluded by trial criteria in the multiple sclerosis RCT. However, there were 3 per year in the TTP trial who did not qualify for plasma infusion and were treated outside the study and at least 2 in total who were treated outside of the study because of age reasons in the myeloma cast nephropathy trial. These numbers would have little effect on the overall increase in activity noted outside the RCT during the period of the RCT. Finally, the increase in activity during the 3 separate RCTs is not merely a reflection of an upward secular trend in the use of apheresis, as noted by a comparison in Figure 1 with apheresis for all conditions.

This jumping-the-gun phenomenon raises perplexing ethical issues that affect physicians and, potentially, pharmaceutical firms that carry out RCTs. Centers involved in RCTs obviously provided apheresis treatment to potentially eligible patients, indicating a compromise of the uncertainty principle. There is no doubt that one of the major problems of RCTs is the potential for misunderstanding of the uncertainty principle and its subsequent compromise in an ongoing study. The problem of allowing patients to confuse experimental intervention with "cutting-edge" therapy seems to be a generic one. Physicians may be guilty of blurring the distinction because they are advocates for an experimental treatment or because they want to encourage enrollment in clinical studies. There did not appear to be any financial incentive differences for physicians within centers outside our trials because the funding was provided by national peer-review agencies or the Ministry of Health; additionally, none of the patients outside the trials went to special apheresis centers set up as for-profit groups because all of the major apheresis centers are affiliated with university programs that are provincially funded. There was little evidence that many patients receiving apheresis outside of the trial failed to meet eligibility criteria for entrance to the trial. Patients cared for in nonparticipating centers may receive experimental therapy as if it were effective treatment but, unlike RCT participants, may not have the same opportunity for informed consent to its use. Not only may patients have been exposed to experimental therapy without appropriate consent, but such treatment jeopardizes and delays findings that would permit optimal therapy.

It has been noted that the publication of results of RCTs provides positive and negative shifts relative to positive and negative results in practice.^11- 12 In the case of lipid-lowering agents, 2 recent inquiries into subsequent use following the publication of major RCTs documented positive shifts in prescribing the lipid-lowering drugs^13- 14 and, in one of the studies, found a significant positive shift in the marketing of the specific drug used in the trial.¹⁴ If this jumping-the-gun phenomenon causes increased use in procedures or drugs used during RCTs, it may result in increased sales of the experimental therapy in both trial and nontrial centers. The motivation for conducting RCTs then would becomes at risk of shifting from a search for efficacy to a search for profits. The centers involved in RCTs obviously provided apheresis treatment to potentially eligible patients, indicating a compromise in the uncertainty principle. There is no doubt that one of the major problems that occurs during the progression of RCTs is the potential for misunderstanding by participating physicians of the uncertainty principle and subsequent compromise in an ongoing study.

Sir George Pickering commented that "once a course of action has been decided upon by tradition, trial or new thinking, it seems there is a license to pile in uncritically ever after."²⁰ We suggest that jumping the gun during the 3 national RCTs reported here is a sign of uncritically piling in during a trial. On a more optimistic note, however, we also observed the rapid discontinuance of useless therapy following a trial that drew a negative conclusion, suggesting that "piling in uncritically" need not be an "ever after" phenomenon.