Eligibility Criteria of Randomized Controlled Trials Published in High-Impact General Medical Journals:  A Systematic Sampling Review FREE

Randomized controlled trials (RCTs) are generally accepted as the most unbiased measures of efficacy for new interventions, drugs, or devices.¹ Results of well-conducted RCTs provide clinicians and health policy makers with the best evidence for adoption or rejection of new therapies.² The findings of RCTs published in major medical journals effect change in medical practice.³ Much attention is paid to the internal validity of clinical trials.⁴ However, even the most well-designed clinical trials are of limited use to clinicians if the results have poor external validity and are not generalizable to the patient population for whom the intervention may be applied.⁴

The external validity of clinical investigations may be threatened by limited patient access to trial centers, selection bias, lack of patient consent to enrollment, and physician resistance to randomization of patients due to treatment preferences^5- 10; however, rigorous inclusion and exclusion criteria may be the greatest challenge to generalizability of results.^11- 16 A recent study demonstrated that among 20 388 Medicare beneficiaries discharged from acute care hospitals in the United States with the principal diagnosis of heart failure, only 13% to 25% met the enrollment criteria of 3 landmark RCTs that have influenced the therapies of all patients with congestive heart failure.¹⁷

Appropriate eligibility criteria are essential for the design of internally valid and efficient RCTs that aim to measure efficacy of an intervention. However, several patient populations, women and children most prominently, have been historically underrepresented in clinical investigations.^18- 28 While it does not make biological sense to enroll women in a prostate cancer trial, it might be reasonable to allow enrollment of males in a breast cancer trial. However, the rationale underlying most exclusion criteria is much less clear. For example, enrollment of women in clinical cardiovascular investigations has been disproportionately smaller than the prevalence of disease among women in the general population.²³ Within handbooks on drug safety and efficacy for children and pregnant and lactating women, the most common directive for common medications is that insufficient information is available within these populations.^25,29- 32 Similar situations frequently exist for certain ethnic groups, the very elderly, and patients with chronic medical illnesses.^{17- 22,29- 30} Advocacy groups and national funding organizations have voiced concern regarding the exclusion of these patient populations from clinical trials.^21,33- 39

The balance between establishing an intervention's efficacy, which may require greater exclusions and a circumscribed patient population, and effectiveness, which necessitates a less restrictive range of participants, is challenging. In the absence of RCT data on those patient populations, clinicians may be compelled to prescribe with trepidation, and gather knowledge about safety and efficacy through anecdotal reports collected outside the structure and safety of clinical trials. Without knowledge of effectiveness, clinicians may offer suboptimal treatment to patients who are underrepresented, or alternatively, expose them to unanticipated harms. The goal of this study was to determine the nature and extent of exclusion criteria among the RCTs published in major medical journals and the contribution of exclusion criteria to the underrepresentation of certain patient populations.

We performed a structured sampling of articles identified through a systematic search of general medical journals with an impact factor of greater than 2.5^40- 42 that published at a minimum of monthly intervals. Using text word or Medical Subject Headings clinical trial; randomized controlled trial; clinical trial, phase I; clinical trial, phase II; clinical trial, phase III; clinical trial, phase IV; and controlled clinical trial, the MEDLINE database was searched for relevant articles. The search was limited to RCTs involving human subjects published between January 1, 1994, and December 31, 2005, in the American Journal of Medicine, Annals of Internal Medicine, Archives of Internal Medicine, British Medical Journal, Canadian Medical Association Journal, Journal of the American Medical Association, Journal of General Internal Medicine, Lancet, and New England Journal of Medicine. More information about the journal search process is available from the authors and at http://www.sunnybrook.ca/research/scientists/AtoF/fowler.

Our search strategy yielded 4827 articles (Figure). Based on findings of a pilot review involving 50 articles from 5 journals published over a 5-year period, we prospectively decided to sample 250 articles published over a 10-year period. Prior to publication, the sample was updated to include 283 articles over a 12-year period (January 1, 2004, to December 31, 2005). This decision was based on the accepted statistical practice of considering only 1 explanatory variable for every 10 outcome events observed in the sample.⁴³ We selected every 16th article from our search, starting from the most recently published. This technique was used to prevent oversampling or undersampling relative to the total number of RCTs published within any 1 journal. Articles were excluded if they were not RCTs or if the trial primarily involved randomization and outcomes of participants or units other than individual patients (eg, physicians or clinics). Each excluded article was replaced with the next RCT published within the same journal, as identified by the MEDLINE search strategy. When identified trials presented long-term follow-up data from earlier reported trials in the selected journals, we confirmed trial characteristics with the original publication. Pilot, phase 1, and phase 2 RCTs are often not intended for immediate translation to patient care but instead are designed to find appropriate doses, ensure intervention safety, or aid with sample size calculations. Such trials were independently assessed by 2 of the authors (H.G.C.V. and R.A.F.) and generally excluded from the final analyses.

Three of the authors (H.G.C.V., A.T., and R.A.F.) independently reviewed all of the articles. The following variables were extracted from each article: publication date; source; corresponding author and country; sponsoring agencies; condition of interest; type of specialty and intervention (eg, drug, device, surgery, or other); university affiliation; single, multicenter, regional, national, or international enrollment; each inclusion and exclusion criterion and where they were listed; number of individuals assessed for eligibility, excluded, randomized, and followed up; and outcomes. When data were incomplete or unclear, we contacted the corresponding author for a complete list of eligibility criteria. The overall methodological quality of each trial was assessed using 4 parameters (generation of the allocation sequence, concealment of treatment allocation, blinding of outcome assessment, and handling of withdrawals and dropouts) that influence the ability of a trial to provide a true estimate of treatment effect.^44- 45

It was necessary to develop definitions that distinguished inclusion from exclusion criteria because ambiguity frequently existed (eg, inclusion criterion: patients with normal renal function; exclusion criterion: patients with renal insufficiency). Inclusion criteria were defined as criteria governing entry or recruitment of individuals into the trial and describing the medical condition of interest. All other criteria limiting the eligibility of individuals were treated as exclusion criteria. For example, among the following inclusion criteria listed in a trial of diabetic foot ulcers: “inpatient in a general medical ward,” “age greater than or equal to 65 years,” “diabetes,” “grade 1 foot ulcers,” and “normal renal function”; only “inpatient in a general medical ward,” “diabetes,” and “grade 1 foot ulcers” were treated as inclusion criteria; age and renal function were analyzed as exclusion criteria.

The reporting of each of the inclusion and exclusion criteria extracted from the articles was classified as strongly justified, potentially justified, and poorly justified. This classification was based on a search of the medical literature on optimal eligibility criteria. Key characteristics were grouped, reduced, and modified. Sensibility testing was performed among a group of clinicians and clinical investigators involved with both clinical trials and observational studies. The classification system was modified until the group was satisfied that it was able to categorize eligibility criteria appropriately and with high-interrater reliability. Exclusion criteria were deemed strongly justified if any specific rationale for the exclusion was provided, even when adjudicators would otherwise have classified the criterion as potentially or poorly justified. Criteria were pilot tested with 50 RCTs before final modification. The final classification criteria were applied without knowledge of other covariates and independently by 2 of the authors (H.G.C.V. and R.A.F.) and compared by the κ statistic. Differences were decided by consensus or independent adjudication (Box).

Strongly Justified Reasons for Excluding Individuals From a Randomized Controlled Trial

Poorly Justified Reasons for Excluding Individuals From a Trial

Potentially Justified Reasons for Excluding Individuals From a Trial

Descriptive data are presented as counts and percentages for categorical variables and mean (SD) or median (interquartile range) for continuous variables. Univariate analyses consisted of 2-sided t test or Mann-Whitney U test for continuous variables and χ² test or Fisher exact test for categorical variables. A Poisson regression model was used to examine the associations between trial characteristics and the number of specific exclusion criteria with estimates displayed by means of risk ratios (RRs) and their associated 95% confidence intervals (CIs). Associations between trial characteristics and the percentage of poorly justified exclusion criteria were examined using a multivariable linear regression model. All statistical tests were 2-tailed. Factors were considered statistically significant at a significance level of P<.05. Due to the inherent heterogeneity among the RCTs selected, quantitative meta-analytic techniques were not appropriate. For all analyses, SAS version 9.1 (SAS Institute Inc, Cary, NC) was used.

The sampled trials represented a variety of medical and surgical subspecialties, sponsorship, academic environments, and scope (Table 1). Evaluation of a medication was the single most common trial focus. Whether the trial was of a medical or surgical patient population, the most common conditions of focus for the trial included infectious disease, cardiovascular disease, cancer, and neurological disorder.

Exclusion criteria were reported in 88.0% of the RCTs, most commonly under the methods section but occasionally within the results section or an appendix. Of the RCTs that did not report exclusion criteria within the text of the publication, only 50% directed the reader to a reference article for exclusion criteria. Age was a reason for exclusion in 72.1% of the RCTs (Table 2). Potential participants younger than 16 years were excluded in 60.1% of trials; those older than 65 years in 38.5% of the RCTs. Restrictions in the inclusion of older adults applied to individuals ranging from age 50 to 80 years.

Conditions related to sex were grounds for exclusion in 47.0% of all trials (Table 2). While females were specifically excluded from 6.7% of trials, conditions associated with female sex were excluded from 39.2% of the RCTs. These included pregnancy (31.9%), lactation (14.5%), lack of contraception use (8.8%), and menopausal status (3.9%). Male sex was the reason for exclusion in only 7.8% of the RCTs.

Individuals were excluded from trial participation due to medical comorbidities in 81.3% of the RCTs, most commonly for kidney, infectious, cardiac, liver, or hematological-oncological disease (Table 2). Many trials (30.9%) did not specify the medical condition that warranted exclusion but listed terms such as “illness,” “comorbidity,” or “significant disease.” Medication-related reasons were the cause of exclusion in 54.1% of trials. Individuals with human immunodeficiency virus or AIDS were excluded from 8.9% of trials, and 12.1% of the RCTs excluded patients on the basis of limited life expectancy. Other common reasons for exclusion included socioeconomic status (13.8%), communication or language barriers (10.6%), participation in other trials (7.1%), and ethnicity (2.1%).

Application of the justification framework by the authors (Box) to the individual exclusion criteria revealed strong agreement (weighted κ, 0.80; 95% CI, 0.78-0.82). A majority (84.1%) of trials contained at least 1 poorly justified exclusion criterion; one quarter of all exclusions were poorly justified in 61.5% of the RCTs (Table 3). Of the 2709 individual exclusion criteria identified, only 47.2% were deemed strongly justified in the context of the individual RCT, 15.9% were graded as potentially justified, and 37.1% as poorly justified. The most common categories of poorly justified participant exclusions were age, medical comorbidity, sex, medication use, and socioeconomic status (Table 3).

Multivariable analyses revealed independent associations between the total number of exclusion criteria and drug intervention trials (RR, 1.35; 95% CI, 1.11-1.65; P = .003) and between the total number of exclusion criteria and multicenter trials (RR, 1.26; 95% CI, 1.06-1.52; P = .009) (Table 4). There was an independent association between the proportion of poorly justified exclusion criteria and trials focused on a medical (vs nonmedical) condition (coefficient estimate, 14.5%; 95% CI, 6.1%-22.9%; P = .001) (Table 5).

Trials with drug interventions (compared with nondrug interventions) were more likely to exclude individuals due to concomitant medication use, medical comorbidities, female sex, and disadvantaged socioeconomic status. Exclusions due to concomitant medication use and medical comorbidities were more likely to be poorly justified (Table 6). Trials of an industry or pharmaceutical-sponsored intervention were more likely to exclude individuals due to concomitant medication use, medical comorbidities, and age. Exclusions due to concomitant medication use and medical comorbidities were more likely to be poorly justified (Table 7).

To investigate whether there was a change in eligibility criteria over time, the RCTs were divided into 2 periods (1994-1998 and 1999-2005), which roughly corresponded to the original publication and dissemination of the CONSORT (Consolidated Standards of Reporting Trials) statements between 1996 and 1998. Compared with the earlier period, the RCTs published between 1999 and 2005 were associated with a nonsignificant improvement in the trial quality score (P = .07), a nonsignificant increase in the mean number of exclusion criteria per trial (8.7 for 1994-1998 vs 10.1 for 1999-2005; P = .05), and a significant reduction in the percentage of poorly justified trial criteria (36.6% for 1994-1998 vs 26.5% for 1999-2005; P = .003).

In this review, we found that the RCTs published in major medical journals may exclude both large segments of the general population and also specific patient populations from the benefits of participation in clinical investigations. We found frequent exclusions of children, the elderly, women (particularly those lactating, pregnant, or able to become pregnant), patients taking common medications, and those with common medical comorbidities. Reasons for exclusions were frequently not justified in the context of individual RCTs. The RCTs with a focus on multicenter drug evaluation were more likely to have a greater number of exclusion criteria. Drug intervention trials and trials with industry sponsorship were associated with exclusions related to comorbidity, medication use, and occasionally age, sex, and socioeconomic status, often without adequate justification.

There are benefits of stringent eligibility criteria. The inclusion and exclusion criteria in a RCT are designed to identify a population of interest in whom an intervention has the greatest likelihood to produce a clinically important and statistically significant effect.³⁸ Efficacy trials with well-defined and homogeneous populations can generally be smaller, shorter, more efficient, and less expensive. This is desirable because clinical trials are increasingly challenged by high costs, limited funding, and regulatory restrictions.³⁹ However, there may be a reciprocal eligibility relationship between efficacy and effectiveness with an inevitable loss of external validity in any tightly designed RCT. The advantages of stringent eligibility criteria are achieved at the risk of excluding patients who may be more likely to represent the population treated in clinical settings and who would better test an intervention's effectiveness.

It is likely that pregnant and lactating women, children, the elderly, and those with medical comorbidities are often excluded from clinical trials out of concern for their safety or the safety of a fetus. However, there is evidence that trial participants assigned to either intervention or placebo have fewer undesirable clinical events and lower mortality rates than those of eligible nonparticipants.^46- 48 Exclusions may also leave future patients with similar characteristics susceptible to unintended harm from an inappropriate generalization of trial results. This was illustrated by the dramatic increase in the rates of hyperkalemia-associated morbidity and mortality following publication of the Randomized Aldactone Evaluation (RALES) trial, largely attributed to the application of trial results to populations that were excluded from the RCT.⁴⁹ There has been similar long-standing prescribing uncertainty for blacks with hypertension due to limited RCT generalizability.⁵⁰ The recent controversies around cyclooxygenase-2 inhibitors may in part stem from narrowly defined patient populations in the original RCTs.⁵¹

Children may be especially susceptible because adverse drug reactions represent a significant cause of pediatric mortality and morbidity.^27- 31 The National Institutes of Health and others have mandated the inclusion of children in all research, barring scientific or ethical reasons.^52- 53 Although initiatives such as the Pediatric Exclusivity Program have improved the translation of knowledge of appropriate and safe prescribing practices to children, our results and those of others indicate that there is still need for careful consideration of age-based exclusions.⁵⁴

The National Institutes of Health has similarly stipulated that research protocols must provide appropriate justification for the exclusion of women from trials and that women should be included in every clinical trial studying a disorder that affects women, irrespective of cost considerations.^33- 36 However, these initiatives have produced only minor changes in the sex composition of cohorts in many areas of clinical investigation.^20,37,55 Underrepresentation of women in clinical trials relative to disease prevalence may partially be due to eligibility criteria that exclude individuals on the basis of lactation, pregnancy, or the potential to become pregnant.

The decision to enroll potentially pregnant or lactating women into RCTs should involve careful consideration of specific risks and benefits of the intervention. Still, many drug and nondrug interventions could be safely studied in this population; inclusion of women in these trials could allow the safety and efficacy to be determined prospectively in a highly monitored setting and could afford increased generalizability to those at risk. Providing safe and effective treatments to pregnant women would promote better outcomes for infants, as recently demonstrated in a pregnancy trial for zidovudine sponsored by the National Institutes of Health, which was terminated early due to a dramatic reduction in perinatal human immunodeficiency virus transmission in the group treated with zidovudine.⁵⁶

The strengths of our study include the volume of RCTs systematically and independently reviewed, which were published in high-impact medical journals between 1994 and 2006. This minimizes the likelihood that our findings are due to selected sampling, chance, or single reviewer bias. Our methods, sample size, and system for justification were refined through an iterative process of pilot testing and modification among clinicians and clinical investigators with expertise in both observational and experimental designs. There was good agreement between reviewers. Finally, we believe our results are compelling.

Limitations of our study should be noted. We chose to focus on exclusion criteria, although there are many factors that may affect the external validity of RCTs, and these may deserve similar attention.⁵⁷ We did not examine journals with a focus on subspecialty populations (for example, pediatrics or obstetrics). However, our purpose was not to examine subspecialty journals that report clinical trials of interventions in specific and narrowly defined populations. Instead, we examined the world's most widely read and quoted general medical journals that publish trials pertaining to conditions of broad interest to physicians and of potential relevance to broad groups of patients. Analysis of trials published in subspecialty journals may have yielded different results. Because we found great variability in the reporting of exclusion criteria among our cohort of RCTs, there may have been underreporting of some exclusion criteria, and our results may represent an underestimation of the true extent of exclusions. Another limitation is that only a minority of trials reported complete participant flow, making it difficult to establish the number of patients eliminated from trial participation on the basis of exclusion criteria alone. However, we attempted to mitigate this by contacting individual authors to obtain such information. Finally, we acknowledge that exclusion criteria are an essential and valuable methodological component of RCT design; however, we have examined the justification of such criteria to determine whether some patients may be unnecessarily excluded from participation. While such a classification is inherently subjective, we have used a prospective and transparent framework on which to base our grading, and independently examined each criterion in duplicate, with a high degree of agreement between the authors.

A perfect balance between efficacy and effectiveness is difficult to achieve within 1 RCT. Distinct RCTs to assess effectiveness might naturally follow preliminary efficacy trials. However, if an intervention has been proven efficacious within an internally valid RCT, it is practically difficult to undertake effectiveness trials because clinical equipoise may no longer exist; unless mandated by a regulatory body, there is limited incentive for industry to fund a RCT when a more modest effect may be expected; and, other agencies rarely have resources to support such a follow-up RCT. Efficacy and effectiveness trials might proceed concurrently within a single large RCT or as separate trials; however, this will also inflate the sample size and cost. The safest and most efficient manner to test interventions may be the current schedule of preclinical work, followed by dose-finding, safety and efficacy trials, with subsequent registries and surveillance systems. Mandatory multiyear registries for patients subsequently treated but who would have been excluded from the original trial is an additional mechanism; yet, this may deter treatment use and be an inferior substitute for sound clinical assessment of the risks and benefits of any therapy.

Based on the results of this study, we recommend that every reasonable effort be made to minimize exclusions of specific patient populations when such patients will likely form a group to which the results are generalized. The CONSORT guidelines and the Uniform Requirements for Manuscripts Submitted to Biomedical Journals have been major steps forward in improving the transparency of RCTs,^45,58 and we did find evidence of fewer unjustified criteria over time; however, further improvements in adherence and reporting can be made.⁵⁷ When exclusions are necessary, we recommend that each criterion be clearly justified within the methods section or appendix. Nonspecific terms such as “medical illness” or “renal insufficiency” should be avoided, opting for more objective criteria (eg, “patients with estimated glomerular filtration rate <30 mL/min”). We recommend that trial inclusion and exclusion criteria be clearly listed within a table that is accessible and apparent to the reader.⁵⁷ This could take the form of a table entitled “who was included and excluded from this trial.” This would not only enable clinicians to more clearly evaluate the generalizability of the results to other patients but also serve as a reminder of the need to consider all groups when designing and reporting clinical trials. Investigators must continue to strive toward making the fruits of clinical research available and relevant to as much of the population as possible: children and adults, females and males, and the sick as well as the healthy.