Limitations of Applying Summary Results of Clinical Trials to Individual Patients: Title and subTitle BreakThe Need for Risk Stratification

There is growing awareness that the results of randomized clinical trials might not apply in a straightforward way to individual patients, even those within the trial. Although randomization theoretically ensures the comparability of treatment groups overall, there remain important differences between individuals in each treatment group that can dramatically affect the likelihood of benefiting from or being harmed by a therapy.^{1 - 4} Averaging effects across such different patients can give misleading results to physicians who care for individual, not average, patients.

The limitations of subgroup analyses—the conventional means for exploring differences in treatment effect based on patient characteristics—are well-appreciated.^{5 - 7} Because patients in trials, as in clinical practice, have many attributes that can affect the likelihood of treatment being beneficial or harmful, exploring each of these attributes “one variable at a time” (eg, male vs female, old vs young) risks spurious false-positive subgroup results from chance fluctuations. Furthermore, although patients have simultaneous multiple characteristics that can affect the likelihood of the outcome and the effect of therapy, one-variable-at-a-time comparisons are fundamentally limited because they compare groups that vary only on a single factor, usually resulting in the subgroups being more similar than different.⁸

For example, consider the benefits that 2 patients with type 2 diabetes mellitus and mild hypertension (blood pressure of 140/90 mm Hg) might get from intensive blood pressure management. Dick is a 65-year-old smoker with a 15-year history of diabetes, a glycated hemoglobin level of 8%, a total cholesterol level of 200 mg/dL (to convert to mmol/L, multiply by 0.0259), and a high-density lipoprotein (HDL) level of 30 mg/dL (to convert to mmol/L, multiply by 0.0259). Jane is a 50-year-old nonsmoker with newly diagnosed diabetes, a glycated hemoglobin level of 7%, a total cholesterol level of 180 mg/dL, and an HDL level of 55 mg/dL. Based on these characteristics, the 5-year risk of cardiovascular death for Dick is more than 20% and for Jane is less than 1%.⁹ If intensive blood pressure control decreased cardiovascular mortality by 10%, that would be important for Dick, but less so for Jane. Indeed, if intensive treatment carried even a small risk of serious harm from polypharmacy, the risks for Jane could easily outweigh the benefits. However, in a clinical trial that tested tight blood pressure control in persons with diabetes, Dick and Jane would be lumped together and the small net risks to patients like Jane would be obscured within the overall average results, overwhelmed by the benefits to patients like Dick.

As another example, consider 2 patients presenting with acute myocardial infarction (AMI). Bud, aged 51 years and otherwise healthy, presents with a small inferior wall AMI, a blood pressure of 164/90 mm Hg, and a stable heart rate. Lou, a 78-year-old patient with diabetes, presents with tachycardia, hypotension (90/60 mm Hg), and a large anterior wall AMI. The acute mortality risk is around 2% for Bud, but 20% for Lou.¹⁰ If thrombolysis yields a 25% relative reduction in mortality, a modest benefit is expected for low-risk Bud, but a substantial benefit is expected for high-risk Lou. With a 1% risk of thrombolytic-related intracranial hemorrhage (and higher in patients with high blood pressure like Bud), treatment decisions should be different for these 2 patients. It would matter little to Bud that the treatment was found to be beneficial overall in the randomized controlled trial when his personal risk of thrombolytic-related harm is more than twice that of his chance of thrombolytic-related benefit.

Risk-based analysis is a statistically powerful tool for uncovering variation of treatment effect in patients like those described above.^{4 ,8 ,11} However, because such risk-stratified analyses are rarely performed, it is not known how frequently substantive differences in net treatment effect remain undetected in clinical trials. In fact, it is not even known how often the summary results of a clinical trial apply to most of the patients in the trial. Counterintuitively, it does not take extreme assumptions to generate conditions where the summary results of a trial do not even apply to the typical patient in the trial. Such conditions are likely to be common and clinical trial results should routinely assess for this by using multivariate risk-stratified analysis.^{4 ,8 ,11}

Large variation in outcome risk between patients in clinical trials appears to be common.¹² For AMI, if patients are stratified using a risk index based on easily obtainable pretreatment variables, the mortality rate in the quartile of patients at highest baseline risk can be more than 10-fold that in the lowest-risk quartile.^{13 - 14} For human immunodeficiency virus trials, outcome rates have been found to be as much as 46 times higher in the highest risk compared with the lowest risk quartile of study participants,¹² and in a pooled sample of trials examining chronic kidney disease progression, this ratio was about 70.¹⁵ When outcome rates differ this much between large subgroups of patients, the degree of benefit and the trade-offs between benefits and harms will necessarily differ substantially across patients even when the treatment has similar relative effectiveness for all.

If baseline risk were normally distributed, the mean benefit for a treatment observed in the trial's summary results would at least reflect the treatment effect found in the typical patient enrolled in the trial. But this is frequently not the case. Often, most trial outcomes occur in a relatively small number of high-risk patients, while most patients are at much lower than average risk.¹⁶ This is because risk variables often cluster in patients. Older patients typically have more risk factors (apart from age) than younger patients and patients with diabetes typically have more risk factors, such as hypertension and dyslipidemias, than patients without diabetes.² Also, since clinical trial outcomes are generally infrequent, a “floor effect” prevents a normal bell-shaped risk distribution, because patients can and will have risks much higher than average but cannot have risks less than 0. Thus, it is common, as for AMI trials, that there are many more low-risk patients (like Bud) than high-risk patients (like Lou). The overall trial results, however, will reflect the arithmetic mean, which is unduly influenced by the relatively small high-risk group. In this type of distribution, the typical (median-risk) patient is at much lower risk than the average (Figure, A).

This effect is most apparent when the treatment-related harm causes primary outcome events (Figure, B and C). There are many such examples. Thrombolytics for AMI can both prevent and cause death, carotid endarterectomy can both prevent and cause stroke, and blood pressure medicines can both prevent and cause cardiac events. However, even when treatments do not adversely affect the main outcome measure of a trial, treatments are virtually always associated with some risk (known or unknown) of adverse effects. Risk-stratified analyses that identify very low-risk patients who get negligible treatment-related benefit are informative even when such patients get the same relative risk reduction as higher-risk patients, because clinicians can at least factor in the potential harms of recommending additional treatments.

A series of simulations⁸ showed that even when (1) a substantial number of study participants receive net harm from a therapy that is beneficial overall, (2) the trial is well-powered, and (3) subgroup analyses are based on common and important risk variables, one-variable-at-a-time subgroup analyses are highly unlikely to detect differences in the treatment effect between subgroups (ie, low statistical power) and will lead to the misleading conclusion that the treatment effect is consistent across patients. This is because a single risk factor will rarely dramatically affect overall patient risk, and thus treatment effect differences are often small in one-variable-at-a-time analyses (Figure, C).

Because of the limitations of conventional subgroup analysis, a common assumption is that there is no alternative. However, a single analysis, stratifying patients by using a multivariate risk model, frequently (although not always) will be sufficiently powered to detect important treatment differences whenever substantial heterogeneity in risk exists within the study population.⁸ This can generally be accomplished with even a moderately predictive multivariate prediction tool,⁸ because a multivariate risk-stratified analysis compares the treatment effect in patients across a fuller spectrum of baseline risk (often ranging in risk by 10 to 30 fold) than a conventional one-variable-at-a-time subgroup analysis (few independent risk factors increase risk by more than 2 fold). Treatment effects are much more likely to differ in patients with greatly different outcome rates (Figure, C). Multiple and varied examples have demonstrated that multivariate risk-stratified analyses can be clinically informative, revealing large subgroups with results that differ substantially from the overall summary.^{13 ,17 - 22}

The key feature of a risk-stratified analysis is that several patient attributes (or risk factors) are combined into a score that describes a single dimension of risk along which treatment effect is likely to vary (almost always on the absolute risk scale, and potentially on the relative risk scale as well). Thus, analyses of this type minimize the major shortcomings of one-variable-at-a-time subgroup analyses (multiple comparisons resulting in false-positive findings and poor statistical power resulting in false-negative findings).

Other methodological issues certainly need to be addressed. Identifying and combining factors that influence additional dimensions of risk, aside from outcome risk, may also be important in determining treatment benefit. These include the risk of treatment-related harm,^{13 ,17 ,21} the risk of competing outcomes unresponsive to therapy, and factors that directly modify treatment effect.⁸ Also, validated multivariate risk indices do not exist for all conditions and outcomes, and those that do exist often show substantially worse performance when tested by independent researchers.

Despite these and other caveats,^{4 ,8} currently there are methods to produce better scientific evidence about the risks and benefits of treatments for patients; however, recommendations continue to be based on crude averages. An important barrier to more individualized therapies is not a paucity of good predictors of outcomes and effects, which are already abundant even in the pregenomic era, but the lack of a consistent analytic approach that informs how an individual patient's multiple attributes combine to affect the fundamental determinants of the desirability of treating that patient—the individual's risk of bad outcomes in the absence of the treatment vs the individual's risk of bad outcomes if treated. Multivariate risk-stratified analyses based on easily obtainable clinical variables are often informative and frequently feasible, but rarely performed. Making such analyses standard should be seriously considered.