Impact of Participant and Physician Intervention Preferences on Randomized Trials: Title and subTitle BreakA Systematic Review FREE

Randomized controlled trials (RCTs) provide the most reliable evidence for treatment efficacy.¹ Although trial participants are often conceptualized as passive recipients of interventions, many have preferences for treatments under evaluation and may decline to consent to randomization. This may limit generalizability of the results to clinical populations (ie, reduce external validity). When treatments cannot be blinded, patients randomly allocated to their nonpreferred intervention may experience resentful demoralization,² which may lead to worse outcomes, either directly (through poor adherence to treatment) or indirectly (through a negative placebo-like effect).³ Thus, preferences may introduce bias (ie, reduce internal validity). Trial designs have been developed to address such problems ( Article ), but their results have not been systematically evaluated.^{4 - 5} There is little consensus on the magnitude of preference effects or the value of information from nonrandomized “preference” cohorts. As patients become more active participants in research, this issue is crucial for continued confidence in the results of randomized trials.

Comprehensive Cohort Design⁴

Two-Stage, Randomized Designs^{5 - 6}

We conducted a systematic review of RCTs that incorporated participants’ preferences to test 2 hypotheses: (1) preferences influence recruitment to trials and thus reduce external validity and (2) preferences influence outcomes in trials and thereby reduce internal validity.

We searched MEDLINE, EMBASE, PsycINFO, CINAHL, AMED, and the Cochrane Library for articles published between 1966 and September 2004. The 2 themes in the first filter of the search strategy were preferences and possible determinants of preferences. The next filter captured all types of randomized trials, meta-analyses, and systematic reviews. We also hand searched BMJ, JAMA, The Lancet, and New England Journal of Medicine; searched reference lists of relevant articles; and contacted authors of published preference designs.

Eligibility. We included comprehensive cohorts and 2-stage trial designs that measured or recorded patient or physician preference; allocated participants to the interventions according to their preferences and by randomization; and followed up randomized and nonrandomized cohorts. We excluded trials with no recording of preference; of decision aids; with measurements of preferences for economic analyses; in which patients who refused randomization were followed up without reference to preferences; and of nonclinical populations. We did not reject trials on the basis of conventional criteria for quality¹ because these relate predominantly to concealment of randomization and were not central to our aim.

Screening Process. We identified 10 023 citations. Two authors independently screened the abstracts of each citation and identified potentially relevant citations. Disagreements were discussed at a steering group. Non–English-language articles were translated for possible inclusion. This screening process yielded 170 articles.

Assessment of Study Eligibility. Up to 4 reviewers independently evaluated the articles, and disagreements were resolved at project steering group meetings. A total of 32 articles were determined to be eligible.

We extracted data on study design, preference measurement, quality of randomization, blinding, and attrition. We collected information on proportion of eligible participants recruited; numbers consenting to randomization; and agreement at each allocation in trials in which participants were allocated to randomization or preference cohorts.^{5 - 6} Primary outcomes were identified through explicit statements, study hypotheses, and reported power analyses. Otherwise, the most likely primary outcome was chosen by consensus (M.K., I.N., and F.L.). We extracted numbers of participants with primary outcome data as well as summary data on the primary outcome(s) at baseline and each follow-up. We examined the trialists’ analysis plans and the results of their analyses. When necessary, we contacted authors for further information. When this was not forthcoming, the trial was included in the review but excluded from our reanalysis. We classified the primary outcomes as subjective or objective. Subjective outcomes were defined as measures of perception of or satisfaction with treatment (directly by self-report or indirectly through clinic attendance); self-reported symptoms; or self-reported behavior. Objective outcomes were those measured independently of the participant such as results of medical tests, mortality, and clinical signs.

Notation. For interventions 1 (experimental) and 2 (control), we represent the randomization groups by R1 and R2 and the preference groups by P1 and P2. In comprehensive cohorts, participants in groups P1 and P2 are those that refused randomization while in 2-stage trials they are those that were randomized to choose their treatment.

Analysis of Preference Effects on Recruitment and Attrition. We examined whether preferences influenced recruitment to trials by examining participation rates and consent to randomization. For comprehensive cohort studies, we examined differences in baseline characteristics between patients who had preferences and those who agreed to randomization by summarizing the numbers and results of reported significance tests.

We examined attrition by ascertaining the percentage of patients lost to follow-up for the primary outcome in the randomization and preference arms of each trial. These percentages were compared between groups using a paired t test (pairing randomization and preference groups within each study) weighted for sample size.

Analysis of Preference Effects on Outcome. We examined treatment-specific differences in outcome between preference and randomization groups (R1 vs P1 and R2 vs P2) for the primary outcome by (1) summarizing the methods and results of reported analyses and (2) conducting a reanalysis using summary data. For (1) we considered all available time points for the primary outcome(s), while for (2) we prespecified a single time point and primary outcome (Table 1 and Table 2).

Because the trials involved a range of disorders, outcome measures, and sample sizes, we converted differences into standardized effect sizes in our reanalysis. We calculated effect sizes directly for continuous outcome variables (difference in means divided by the pooled standard deviation) and calculated log odds ratios for binary outcomes before converting them into approximate effect sizes by dividing by 1.81.⁴⁹ Outcome effect sizes were categorized using absolute values as |0.20| or lower; |0.21| to |0.50|; and greater than |0.50|.⁵⁰ The level of significance was set a priori at P≤.05. Data were analyzed using Excel (Microsoft, Redmond, Wash) and SPSS version 11.5 (SPSS Inc, Chicago, Ill).

We emphasize that comparisons of outcome between randomization and preference groups are subject to bias, and we were not able to adjust for confounding factors in calculation of effect sizes. Therefore, for trials with baseline measures of the outcome variable, we conducted an additional analysis in which we presented baseline and outcome effect sizes and the “net” effect size (outcome minus baseline). These trials provide the strongest evidence for preference effects on outcome.

Thirty-two trials (27 comprehensive cohort trials^{7 - 43} and 5 two-stage randomized designs^{3 ,44 - 48} ) were identified (Table 1 and Table 2). The studies covered a range of clinical areas and interventions. The information on interventions given to participants and the method of elicitation of intervention preferences varied widely. In 5 comprehensive cohort studies, physician preference was reported as a factor in the decision to decline randomization or in the final choice of treatment (Table 1).^{13 ,21 - 24 ,32 - 33 ,38 - 39} Parental preference was relevant in 4 studies involving children.^{34 - 36 ,38 - 39}

Trial Recruitment and Acceptance of Randomization. Eleven of the 27 comprehensive cohort trials did not report the number of eligible individuals who agreed to participate (ie, in either randomization or preference groups). Among the remaining 16 studies, the participation rate ranged from 50% to 100%; 12 of 16 studies reported a participation rate higher than 80% and 9 reported higher than 95% participation. Among individuals who agreed to participate, acceptance of randomization in all 27 comprehensive cohort trials varied from 26% to 88% and was lower than 50% in 14 of 27 studies. In 2 of the 5 two-stage trials, participants were randomized (to choose the intervention or undergo a second randomization) prior to consent.^{3 ,46} Two studies provided no information^{44 - 45 ,47} and in one trial 62% of eligible participants agreed to be randomized at the first stage.⁴⁸ Participation in the next stage (allocation to R1 or R2 and P1 or P2) ranged from 48% to 100% agreeing to choose a treatment in the preference groups and 25% to 70% accepting randomization.

Baseline Differences Between Randomization and Preference Cohorts. Twenty of the 27 comprehensive cohort designs reported at least one comparison between randomization and preference groups on baseline sociodemographic factors. At least one significant difference between randomization and preference groups was found in 9 of the 20 trials. Fifteen significant differences were reported in 89 sociodemographic comparisons, a finding greater than that expected by chance. The proportion of significant findings did not differ greatly according to sample size (7/38 and 8/51 in comparison with total sample size <300 and ≥300, respectively). Significant sociodemographic findings were as follows: compared with participants accepting randomization, those with an intervention preference were more likely to be well educated (4/9 trials tested),^{13 ,22 - 24 ,28 ,38 - 39} employed (3/9 trials tested),^{28 ,34 ,42} white (2/9 trials tested),^{13 ,28} women (in 1 trial),⁴² men (in 1 trial; 12 tested for sex),²⁰ unmarried (1/5 trials tested),^{25 - 27} less likely to have siblings (in 2 trials involving children),^{34 ,38 - 39} and more likely to be in day care (1 trial).^{38 - 39}

Twenty-two of 27 comprehensive cohort trials reported at least one comparison between randomization and preference groups on baseline clinical and health characteristics. At least one significant difference was reported by 8 of the 22 trials. Overall, 29 significant differences were found in a total of 260 clinical comparisons, a finding slightly greater than that expected by chance. The proportion of significant findings for clinical characteristics was higher among larger studies (4/92 and 25/168 in comparisons with total sample size <300 and ≥300, respectively). Of the 8 trials that reported significant differences, participants in the preference group had more severe clinical problems in 2^{20 ,25 - 27} and less severe clinical problems in 3,^{13 ,28 - 29} while in the remaining 3 trials there was no consistent pattern.^{21 - 24 ,32 - 33}

Attrition. Authors provided information on attrition in both randomization and preference groups in 29 of 32 trials. Among these 29 trials, the proportion of individuals with missing data for the primary outcome was similar for randomization and preference groups. For randomization groups, the numbers of studies missing data were as follows: 0% to 9% missing data, 18 trials; 10% to 19%, 7 trials; and 20% or more, 4 trials. The corresponding numbers in the preference groups were 17, 8, and 4. The mean percentage of participants lost to follow-up was 10.5% and 8.8% for randomization and preference groups, respectively (paired t test weighted for sample size, P = .62).

Analysis of Preference and Outcome Reported By the Trialists. Nineteen of the 32 comprehensive cohort and 2-stage studies examined preference effects on outcome by performing treatment-specific or treatment-adjusted comparisons of randomization and preference groups for at least one primary outcome measure and follow-up point.^{3 ,9 - 12 ,14 ,16 - 18 ,20 - 24 ,28 - 30 ,32 - 33 ,38 - 39 ,42 - 48} Adjustment was made for baseline disease and/or other characteristics in only 9 of the 19 studies.^{3 ,11 - 12 ,21 - 24 ,28 - 29 ,46 - 48} Seven of the 19 studies reported at least one statistically significant difference between preference and randomization groups at outcome.^{3 ,11 - 12 ,16 - 18 ,20 ,22 - 24 ,29 ,44 - 45} In 5 of the 7 studies, the preference group was favored,^{3 ,12 ,16 - 18 ,21 - 24 ,44 - 45} while in the remaining 2 studies, the randomization group was favored^{20 ,29} ( Article

Outcome comparisons in which the preference group had more favorable outcome:

Outcome comparisons for which the randomized group had more favorable outcome:

Reanalysis Using Standardized Effect Sizes. We derived standardized intervention-specific preference effects (based on the difference between randomization and preference groups at outcome) using a single outcome and follow-up time for each study (Table 1 and Table 2). We were able to calculate standardized effect sizes in 25 of 32 studies. Of the 7 remaining studies, 5 provided no summary data for the preference arms,^{15 ,40 - 42 ,48} while standard deviations were not given and could not be approximated in 2.^{20 ,30} Of the 25 studies for which we could calculate preference effects, 2 provided only pooled data for randomization and preference groups^{14 ,47} and 2 had only one intervention group for which a randomization vs preference comparison was applicable,^{9 - 10 ,46} leaving a total of 46 randomization-preference comparisons.

Preference and Outcome Ignoring Baseline DifferencesTable 3 shows the standardized effect sizes for these 46 comparisons listed by sample size and classified according to magnitude of preference effect size and type of outcome (subjective or objective). Of the 46 comparisons, only 5 (from 5 studies) were statistically significant (Table 3), all of which favored the preference arm.^{3 ,11 - 13 ,19 ,44 - 45} Three of these 5 comparisons were reported as significant in the investigators’ analyses ( Article ).^{3 ,11 - 12 ,44 - 45} The remaining significant effects (not reported by investigators) were for change in body mass index in the group not receiving hormone therapy in a study of its use for postmenopausal women¹⁹ and cardiac mortality in the group receiving PTCA comparing it with CABG in diabetic patients with coronary disease.¹³ Four significant randomization vs preference comparisons from the investigators’ analyses were not observed in our reanalysis. These 4 cases are explained by omission from our reanalysis because of insufficient infomation²⁰ ; our prespecification of a single primary outcome (multiple outcomes were tested in the investigators’ analysis)^{16 - 18} ; and our analysis ignoring baseline covariates.^{22 - 24 ,29}

We evaluated preference effects according to magnitude of effect size. The majority of effect sizes for the 46 comparisons were small: 25 (54.3%) from 18 studies were |0.20| or less, 13 (28.3%) from 10 studies were between |0.21| and |0.50|, and 8 (17.4%) from 7 studies were higher than |0.50|. Of the 8 effect sizes higher than |0.50|, 6 favored the preference and 2 the randomization groups (2 occurred in a single study but were in opposing directions for the intervention groups^{44 - 45} ). Furthermore, moderate or large effect sizes were much less likely to occur in large studies compared with smaller ones (effect size was >|0.20| for 33% [9/27] of comparisons based on ≥100 participants vs 63% [12/19] of comparisons based on <100 participants). There was little evidence of variation in effect size according to the objectivity of outcome measure: effect size was greater than |0.20| for 44% (7/16) of comparisons with objective outcomes compared with 47% (14/30) of comparisons with subjective outcomes. When we restricted this comparison to effect sizes based on 100 or more participants, the corresponding percentages were 25% (3/12) for objective outcomes vs 40% (6/15) for subjective outcomes.

Preference and Outcome After Accounting for Baseline Differences. In 10 of the 25 studies for which we could calculate standardized effect sizes (18 of 46 intervention-specific comparisons), we were able to evaluate preference effects after accounting for baseline differences in the primary outcome variable between randomization and preference groups. In 9 studies we could derive baseline effect sizes for the relevant outcome variable,^{8 ,11 - 12 ,16 - 18 ,21 ,25 - 29 ,36 ,46} while for the 10th a change from baseline measure was used as an outcome in itself.¹⁹ Only 2 of the 18 comparisons were based on objective outcome measures. Figure 1 shows baseline and outcome effect sizes for the 18 intervention-specific comparisons in order of sample size (smallest at the top of the figure). The proximity of the open and closed boxes for each comparison demonstrates the outcome effect size in relation to the baseline effect size. For most comparisons, the direction and magnitude of the baseline and outcome effect sizes were similar, suggesting no clear effect of preference on outcome over and above differences that existed at baseline. Net preference effect sizes (outcome minus baseline) are shown in Figure 2. Most net effect sizes were small: 12 (66.7%) from 9 studies were |0.20| or less, 4 (22.2%) from 3 studies were between |0.21| and |0.50|, while 2 (11.1%) from 2 studies were higher than |0.50|. The 2 large net effect sizes both favored the randomization group.^{28 - 29} Figures 1 and 2 show that the studies with largest sample sizes (which give the most precise estimates) tended to have the smallest baseline, outcome, and net preference effect sizes, which were clustered around zero.

Our findings indicate that participants’ preferences affect recruitment. In many trials, substantial numbers refused randomization because of preferences. There was some indication that the more “empowered” (well educated and employed) are more likely to refuse randomization because of preferences. Differences in clinical characteristics showed no consistent pattern in favor of randomization or preference groups, and sociodemographic or clinical differences were evident in only a minority of studies. Thus, although preferences reduced recruitment, it was less convincing that external validity was seriously compromised, although differences may occur in important unmeasured variables.

With regard to our second hypothesis, there was some evidence that participant or physician preferences influenced outcome in a proportion of trials. However, evidence for moderate or large preference effects was much weaker in large trials and after accounting for baseline differences. Where preference effects were evident, they were inconsistent in direction and were not clearly associated with whether the primary outcome was subjective or objective. There was no evidence that preferences influenced attrition. Therefore, although our review confirms that patients and physicians often have strong intervention preferences, it gives less support to the hypothesis that preferences significantly compromise internal validity.

Patient preference trials provide unique data on external and internal validity. However, there are a number of limitations to our review. Interventions and settings were diverse, which limited the extent to which we could pool data from individual studies. Twenty-seven of the 32 trials were comprehensive cohort designs and thus we can make only limited conclusions about 2-stage preference designs. Preference effects, however, might be expected to be strongest in comprehensive cohorts because they include people who refuse randomization. Seven trials did not contain sufficient data for us to calculate effect sizes. Assessment of the effect of preference on outcome is subject to confounding. The extent to which it was possible to account for this was limited. To be comprehensive, we sought studies from as far back as 1966, so studies were of variable quality. In reporting the results of investigators’ published analyses we cannot rule out the possibility that comparisons were undertaken but not published and that significant findings were more likely to be reported than nonsignificant ones. However, such a bias would result in our overestimating the importance of preference in baseline and outcome comparisons.

Our review also provides insight into the outcome of nonrandomized participants who are followed up under the same experimental conditions as are randomized participants. Thus it adds to the growing evidence that where strong preferences or ethical objections to an RCT exist, observational methods are a valuable alternative.^{51 - 52} Although RCTs remain the gold standard, the outcomes of nonrandomized cohorts best approximate to those of randomized studies when exclusion criteria are carefully defined and when prognostic factors and patients’ and professionals’ preferences are well understood. These methods could be used to investigate the effectiveness of interventions when randomization is unacceptable to clinicians and/or patients and their families.⁵³

We conclude that intervention preferences appear to have limited impact on the external or internal validity of randomized trials.